CONCEPT
Selective Reporting
The practice of publishing the benchmark scores on which a model does well and omitting the ones on which it does not — a simple form of Goodhart's Law gaming that survives because no party has the standing to enforce disclosure.
Selective reporting is the systematic publication bias that results when labs are free to choose which benchmarks accompany a release. Every frontier model is run on many more evaluations than appear in its release announcement. The ones that appear are, with high reliability, the ones that flatter the model. The ones that are omitted are often the ones that would most inform a prospective user. This is not fraud; it is marketing, and it is the default behavior of any institution under commercial pressure. It nonetheless produces a comparison landscape in which every announced model looks better than every previous model on every measure, which cannot be collectively true.
In The You On AI Field Guide
The asymmetry is visible in release after release. A new model's press materials report the benchmarks on which it exceeds the previous generation or the nearest competitor. Benchmarks on which it merely matches are often excluded. Benchmarks on