CONCEPT
Prompt-Template Hacking
Small, non-semantic adjustments to how a benchmark is presented to a model — phrasing of the instruction, ordering of options, use of chain-of-thought, formatting of the answer — that move reported scores by tens of points without changing what the model actually knows.
Prompt-template hacking is the practice of tuning the exact template in which a benchmark is presented to a model in order to maximize the reported score. Because language models are sensitive to surface form — a ten-word change in instruction can shift scores by 10–20 percentage points — and because benchmark authors typically do not specify an exact template, labs have wide latitude in how they run the evaluation they report. Over time this has produced a pervasive gap between "the score we can get with our best template" and "the score the model would get under a fair, author-specified template." Where the two numbers are allowed to diverge, benchmark leaderboards become a measurement of prompt-engineering skill as much as of model capability.
In The You On AI Field Guide
The most studied instances are multi-choice benchmarks. A model presented with a four-option question can be scored in at least three different