Scaling Laws

In The You On AI Encyclopedia

The Kaplan paper's headline finding was that language-model loss decreases as a power law in compute, parameters, and dataset size — and that the exponents of these power laws are stable across model sizes and architectures. The result established that scale is a first-order input to capability, not an engineering detail. It also established that the field could, for the first time, make forecasts grounded in measured regularities rather than expert opinion.

Chinchilla (Hoffmann et al., 2022) revised Kaplan's prescriptions: optimal training requires roughly 20 tokens per parameter, not the parameter-heavy regime the Kaplan paper suggested. The revision was consequential — it explained why older models like GPT-3 (which were parameter-rich but data-starved) underperformed newer models trained on more tokens. Chinchilla-style recipes have dominated frontier pretraining since, with the ratio later pushed to 100+ tokens per parameter by Llama-3 and related work.

The scaling laws are the reason for the present AI investment cycle. A company deciding whether to spend two billion dollars on a training run is making a bet grounded in the laws' forecast that the resulting model will be meaningfully more capable than its predecessor. This is a radical empirical claim — no other technology sector has ever had quantitative capability forecasts this reliable. The risk is that the laws are limited to a regime (text, roughly transformer architectures, standard data) and that the regime is ending. Emergent capabilities complicate the picture further by introducing step-function gains the smooth loss curves do not predict.

The forecasting utility of the laws is uneven. They predict loss extremely well; they predict downstream capabilities (benchmark scores, reasoning, code generation) less well; they predict economic impact worse still. The current frontier of the field is determining whether the next order of magnitude in compute will continue to produce the capability gains the laws project — and whether, when the laws break, the break will be gentle (sub-linear returns to compute) or sharp (qualitative new phenomena). The answer will shape the next decade of AI trajectory.

Origin

The scaling-law research program began with Hestness et al.'s Deep Learning Scaling is Predictable, Empirically (2017). Kaplan et al.'s Scaling Laws for Neural Language Models (2020) generalized the approach to transformer LMs. Hoffmann et al.'s Training Compute-Optimal Large Language Models (2022) corrected the Kaplan recipe. Subsequent work by Anthropic, DeepMind, and independent researchers has extended the laws to multi-modal settings, reasoning benchmarks, and post-training stages.

Key Ideas

Power laws are the regularity. Loss decreases as a power of compute, parameters, and data — five-order-of-magnitude stability makes this the most robust empirical law in the field.

Optimal allocation is the operational insight. For a given compute budget, the Chinchilla ratio (tokens ≈ 20× parameters, now higher) guides frontier training.

Capability is not directly predicted. Loss scales cleanly; downstream capabilities scale less cleanly; economic impact barely at all.

The regime-end is the frontier uncertainty. Whether continued scaling produces continued gains, and if not how the transition unfolds, is the highest-leverage open question.