CONCEPT

The Experimenter's Regress

Collins's 1985 demonstration that scientific controversies <em>cannot be resolved by experimental results alone</em> — because the only way to know whether an experiment was performed correctly is to know its expected outcome, which is the thing the experiment was designed to determine.

The experimenter's regress is one of Collins's foundational contributions to the sociology of scientific knowledge, developed in Changing Order (1985) through his study of the gravitational wave physics community. The argument is structural: when a novel experiment fails to produce the expected result, there are always two possible explanations — the theory is wrong, or the experiment was performed incorrectly. But the only way to distinguish these possibilities is to know the correct result in advance, which is precisely what the experiment was designed to establish. The regress cannot be broken by logic. It is broken by social negotiation — by the community's judgment about which experimenters are competent, which equipment is reliable, and which results to trust.

In The You On AI Field Guide

The regress has direct implications for AI evaluation. When a language model produces output in a specialized domain, evaluating its correctness requires the same kind of social judgment that the experimenter's regress demands. The model's output either conforms to the expected form of expert discourse or it does not, but 'expected form' is itself a product of the community's ongoing social negotiations about what counts as competent practice. A non-expert cannot break the regress on their own. They must either accept the community's judgment or participate in the community's practices long enough to develop their own.

The parallel to collective tacit knowledge is direct. The regress-breaking judgments are maintained in social practice, not in published protocols. A machine trained on the community's published output can reproduce the form of the judgments but not their substance, because the substance lives in the community's life rather than its text.

Origin

Collins developed the argument through his study of Joseph Weber's controversial gravitational wave detection claims in the 1970s. The community's process of evaluating and ultimately rejecting Weber's results revealed the impossibility of purely experimental resolution and made visible the social processes through which scientific consensus is actually achieved. Changing Order (1985) consolidated the argument into a general framework.

Key Ideas

Structural. The regress is not a flaw in scientific practice but an inherent feature of empirical inquiry at the frontier.

Socially broken. Communities break the regress through judgments about competence, reliability, and credibility — judgments maintained in collective tacit knowledge.

AI relevance. Evaluating AI output faces the same structural challenge: without independent access to expert judgment, the evaluator cannot distinguish correct from fluently wrong output.

The replication problem. The regress explains why scientific results cannot be reliably replicated from published protocols alone — the tacit knowledge required is social.

Explore more

Browse the full You On AI Field Guide — over 8,500 entries