CONCEPT
The Experimenter's Regress
Collins's 1985 demonstration that scientific controversies cannot be resolved by experimental results alone — because the only way to know whether an experiment was performed correctly is to know its expected outcome, which is the thing the experiment was designed to determine.
The experimenter's regress is one of Collins's foundational contributions to the sociology of scientific knowledge, developed in Changing Order (1985) through his study of the gravitational wave physics community. The argument is structural: when a novel experiment fails to produce the expected result, there are always two possible explanations — the theory is wrong, or the experiment was performed incorrectly. But the only way to distinguish these possibilities is to know the correct result in advance, which is precisely what the experiment was designed to establish. The regress cannot be broken by logic. It is broken by social negotiation — by the community's judgment about which experimenters are competent, which equipment is reliable, and which results to trust.
In The You On AI Field Guide
The regress has direct implications for AI evaluation. When a language model produces output in a specialized domain, evaluating its correctness requires the same kind of