The Tyranny of Demonstrations

The tyranny of demonstrations is Rodney Brooks’s name for the most reliable cognitive trap in the history of artificial intelligence: the error of watching a system perform impressively on a curated task and concluding that it possesses broad, general, deployable competence. A demonstration is designed, consciously or not, to show a system at its best in conditions chosen to flatter it. The presenter selects the task, controls the environment, and edits the failures. The audience, applying inference heuristics evolved for judging other humans, generalizes from the performance to a surrounding competence that the system does not in fact possess—because, unlike human competence, machine competence does not bundle in the predictable ways that license such inference. A human who can read a menu can read a novel; a language model that writes a fluent essay may fabricate citations with equal fluency and equal confidence, because both acts are formally identical to it. Fluency decorrelates from authority in machines as