A scaling law is an empirical regularity, distilled from experiments, describing how a model's training loss — roughly, its prediction error — falls as three quantities grow: parameter count, training data volume, and compute spent. The key finding is that the decline is smooth and predictable, following a power law across a wide range.
The significance is more engineering than theory. It means you can run a family of small models, fit the curve, and extrapolate where the loss will land if you spend ten times the compute. The nine-figure training runs of recent years rest on that: the curve made the investment estimable rather than a pure bet.
Grab It in One Sentence First
Scaling laws say model quality doesn't come from a flash of insight — it improves steadily along a curve you can compute in advance as parameters, data, and compute increase.
An analogy: if you observe that every doubling of a plant's capacity cuts unit cost by a fixed proportion, you can work out the most economical plant size before breaking ground, instead of building one of each. Scaling laws give large-model training that same ability to price a decision beforehand.
Two Key Results
The first (Kaplan et al., 2020). Loss follows power laws in parameters, data, and compute, with no visible breaking point across a wide range. It carried a then-counterintuitive corollary: at fixed compute, training a large model short of convergence often beats training a small model to convergence.
The second (Chinchilla, 2022). The first result underweighted data. After large-scale experiments, DeepMind refit the relationship and found that for a given compute budget, parameters and training tokens should grow roughly in proportion. By that result, many models of the era were over-parameterized and under-fed — the same compute spent on a smaller model trained on more data would have done better.
Chinchilla changed industry practice directly: later models generally stopped racing on parameter count and put more of the budget into data volume and data quality.
From Training Scaling to Inference Scaling
Early scaling laws were all about the training side. The last couple of years added a second axis: test-time compute. With the same model, generating a longer chain of thought, sampling several paths in parallel and picking the best, or checking itself repeatedly all raise accuracy on hard problems as inference compute grows.
That axis changed the cost structure. Compute used to be spent once in training and then amortized across cheap inference; now hard queries consume real compute at inference time, which is why reasoning models price and behave differently from ordinary ones.
The Ceiling on the Data Side
Data is the hard constraint in a scaling law, and high-quality text is not unlimited. A few practical consequences follow.
- Data quality carries more weight. At the same token count, a deduplicated, cleaned, well-mixed dataset performs noticeably better.
- Synthetic data becomes an important supplement, especially for code, mathematics, and robotics or physical settings where real data is scarce. Training on model-generated data demands strict quality control, or errors compound.
- Multimodality is one route to more data: images, video, and audio still hold a great deal of underused signal.
Where People Get It Wrong
"Falling loss means a better product." Scaling laws predict training loss, not the business metric you care about. Loss correlates with downstream capability but isn't equivalent to it; two models on the same loss curve can differ sharply in instruction following, safety, and tool use, which come mostly from post-training.
"A power law means unlimited improvement." The other face of a power law is diminishing returns: each further increment demands exponentially more investment. And the law itself is an empirical fit with no guarantee of holding outside the fitted range.
"Scaling laws say bigger is better." Rather the opposite — the post-Chinchilla reading is "balance within a budget." And deployment has entirely different goals: small language models, quantization, and distillation all aim to cut cost to the minimum that still works.
"Scaling guarantees a path to AGI." This is the most contested claim. What scaling reliably improves is next-token prediction; whether that is equivalent to steadily improving reasoning, planning, and real-world reliability remains unsettled — see AGI.
What It Means for You
You don't need the formulas, but two judgments are practical. First, the gap between model generations usually exceeds the gap between vendors within a generation: rather than agonizing over which model in a tier, confirm you're on the current tier at all. Second, parameter count is no longer a useful comparison: an MoE model's total and active parameters are different things, and neither training data volume nor post-training quality appears on a spec sheet. Measured results and your own evals beat published numbers.