Benchmark Saturation

In the AI Story

The saturation curve for a typical benchmark has become predictable. A benchmark is released with a state-of-the-art around 50–60%. Within eighteen months, a frontier model passes 80%. Within three years, the leading score is above 90% and the gap between top systems has narrowed to the point where benchmark-specific noise exceeds the differences. The remaining ceiling is often imposed not by model capability but by errors in the benchmark's own ground truth: MMLU has several hundred questions whose canonical answers are contested or wrong, GSM8K has a handful of arithmetic errors in its gold solutions, HumanEval tests have accepted multiple correct implementations. Once models cross 95%, further reported gains are indistinguishable from measurement noise.

Saturation interacts with training-data contamination and post-training specialization in compounding ways. Contamination raises scores artificially, which accelerates apparent saturation and advances the date at which the benchmark becomes useless. Specialization concentrates optimization on the benchmark's specific distribution, which further compresses the discriminative range. A benchmark that might have remained informative for five years saturates in two.

The response from the research community has been to build new benchmarks at accelerating pace. MMLU-Pro, GPQA, HLE (Humanity's Last Exam), ARC-AGI-2, SWE-Bench Verified, LiveCodeBench, FrontierMath. Several are deliberately structured to resist gaming: private evaluation sets, human-verified grading, problems written by domain experts to require reasoning rather than recall. Several are explicit attempts to build a benchmark that the present generation of models fails decisively, precisely so that progress can be observed. This is a reasonable response and a Sisyphean one: every new benchmark is itself a Goodhart target the moment it becomes important enough to report.

For users of AI systems, the practical consequence of saturation is that announced benchmark scores in 2025 carry much less information than they did in 2021. A headline of "89% on MMLU" is roughly as informative as "runs on a computer." The scores that carry information are the ones on newly released, held-out, or private benchmarks, and the margins there are smaller. A literate consumer of model releases now asks: which benchmarks are saturated, which are contaminated, which are specialized-against, and what fraction of the reported number is real capability versus gaming.

Origin

The pattern of benchmarks being "beaten" was noted in computer vision starting with ImageNet, whose top-5 accuracy passed human baselines around 2015 and has since become a compliance check rather than a frontier. In NLP, GLUE was "solved" within two years of its 2018 release, which prompted its authors to release SuperGLUE (2019), which was substantially beaten within another two years. The term "benchmark saturation" became common in LLM discourse around 2022–2023 as MMLU — intended as a long-horizon benchmark — was approached more quickly than anticipated.

Key Ideas

The ceiling is often not real capability but ground-truth error. Above 95%, the benchmark's own mistakes dominate the remaining gap.

Saturation accelerates with contamination. Each public release that includes benchmark analysis feeds the next model's training corpus.

Discrimination loss, not progress loss. A saturated benchmark is not wrong about what is possible; it has stopped being useful for comparing two systems that are both capable.

The arms race is structural. New benchmarks are built, become important, get gamed, saturate, and are replaced. The cycle is shortening.