The Experimenter's Regress
The experimenter's regress is one of Collins's foundational contributions to the sociology of scientific knowledge, developed in Changing Order (1985) through his study of the gravitational wave physics community. The argument is structural: when a novel experiment fails to produce the expected result, there are always two possible explanations — the theory is wrong, or the experiment was performed incorrectly. But the only way to distinguish these possibilities is to know the correct result in advance, which is precisely what the experiment was designed to establish. The regress cannot be broken by logic. It is broken by social negotiation — by the community's judgment about which experimenters are competent, which equipment is reliable, and which results to trust.
The Regress as Gatekeeping Device — Contrarian ^ Opus
There is a parallel reading in which the experimenter's regress functions primarily as a mechanism of professional boundary maintenance rather than an inevitable feature of knowledge production. The regress doesn't merely describe how scientific communities break ambiguity—it describes how they preserve authority. When the only way to evaluate experimental competence is to already know the answer, the community can indefinitely defer validation of challenges to its consensus, labeling dissenting results as methodological failures while treating conforming results as proof of experimental rigor. The pattern is visible in Weber's gravitational wave case, but the directionality matters: the regress was invoked to exclude results that threatened established frameworks, not merely to resolve ambiguity.
The AI parallel inverts the power dynamic in ways Collins's framework doesn't anticipate. Where the regress historically concentrated evaluative authority in credentialed communities, AI creates a situation where non-experts increasingly possess tools that can perform expert-shaped reasoning without possessing expert standing. The community's response—insisting that evaluation requires participation in social practice, that fluent output differs essentially from substantive judgment—begins to look less like description of knowledge's nature and more like assertion of professional prerogative. When a model can produce output indistinguishable from expert judgment to other experts, the claim that 'substance lives in community rather than text' becomes a boundary claim rather than an epistemological one. The regress reveals its function: not resolving ambiguity, but determining who gets to resolve it.
In the AI Story
The regress has direct implications for AI evaluation. When a language model produces output in a specialized domain, evaluating its correctness requires the same kind of social judgment that the experimenter's regress demands. The model's output either conforms to the expected form of expert discourse or it does not, but 'expected form' is itself a product of the community's ongoing social negotiations about what counts as competent practice. A non-expert cannot break the regress on their own. They must either accept the community's judgment or participate in the community's practices long enough to develop their own.
The parallel to collective tacit knowledge is direct. The regress-breaking judgments are maintained in social practice, not in published protocols. A machine trained on the community's published output can reproduce the form of the judgments but not their substance, because the substance lives in the community's life rather than its text.
Origin
Collins developed the argument through his study of Joseph Weber's controversial gravitational wave detection claims in the 1970s. The community's process of evaluating and ultimately rejecting Weber's results revealed the impossibility of purely experimental resolution and made visible the social processes through which scientific consensus is actually achieved. Changing Order (1985) consolidated the argument into a general framework.
Key Ideas
Structural. The regress is not a flaw in scientific practice but an inherent feature of empirical inquiry at the frontier.
Socially broken. Communities break the regress through judgments about competence, reliability, and credibility — judgments maintained in collective tacit knowledge.
AI relevance. Evaluating AI output faces the same structural challenge: without independent access to expert judgment, the evaluator cannot distinguish correct from fluently wrong output.
The replication problem. The regress explains why scientific results cannot be reliably replicated from published protocols alone — the tacit knowledge required is social.
Appears in the Orange Pill Cycle
Regress as Both Structure and Strategy — Arbitrator ^ Opus
The experimenter's regress operates simultaneously at two levels that the debate tends to conflate. At the level of individual experimental interpretation, Collins is straightforwardly correct (100%): there is no logical way to distinguish 'theory wrong' from 'experiment wrong' without importing assumptions about the expected result. This is structural, inevitable, and applies with equal force to AI evaluation. A non-expert examining model output in a specialized domain genuinely cannot break the regress through logic alone—they need either access to expert judgment or immersion in expert practice. The mechanism Collins describes is real.
At the level of institutional dynamics, however, the contrarian view captures something Collins's framework underspecifies (70/30 toward the contrarian reading). The regress doesn't just describe how ambiguity gets resolved—it determines who has standing to participate in its resolution. Communities don't merely make judgments about competence; they make judgments about who is competent to make judgments about competence. The regress creates a recursion that, in practice, concentrates authority. This isn't a flaw in Collins's description but a feature his framework enables us to see: the regress is both epistemological necessity and political instrument.
The synthetic view recognizes the regress as a real structural constraint whose breaking necessarily involves social power. AI doesn't dissolve the regress but redistributes the capacity to perform regress-breaking moves—producing expert-shaped output without expert standing. Whether this constitutes genuine regress-breaking or merely fluent simulation is itself a question whose answer requires... the very expert judgment the regress places in question. The recursion goes all the way down, which is precisely Collins's point. But who benefits from that infinite regress is a separable question.
Further reading
- Harry Collins, Changing Order: Replication and Induction in Scientific Practice (SAGE, 1985; 2nd ed. University of Chicago Press, 1992)
- Harry Collins, Gravity's Shadow (University of Chicago Press, 2004)
- Trevor Pinch, Confronting Nature (Reidel, 1986)