The Experimenter's Regress
In The You On AI Field Guide
The regress has direct implications for AI evaluation. When a language model produces output in a specialized domain, evaluating its correctness requires the same kind of social judgment that the experimenter's regress demands. The model's output either conforms to the expected form of expert discourse or it does not, but 'expected form' is itself a product of the community's ongoing social negotiations about what counts as competent practice. A non-expert cannot break the regress on their own. They must either accept the community's judgment or participate in the community's practices long enough to develop their own.
The parallel to collective tacit knowledge is direct. The regress-breaking judgments are maintained in social practice, not in published protocols. A machine trained on the community's published output can reproduce the form of the judgments but not their substance, because the substance lives in the community's life rather than its text.
Origin
Collins developed the argument through his study of Joseph Weber's controversial gravitational wave detection claims in the 1970s. The community's process of evaluating and ultimately rejecting Weber's results revealed the impossibility of purely experimental resolution and made visible the social processes through which scientific consensus is actually achieved. Changing Order (1985) consolidated the argument into a general framework.
Key Ideas
Structural. The regress is not a flaw in scientific practice but an inherent feature of empirical inquiry at the frontier.
Socially broken. Communities break the regress through judgments about competence, reliability, and credibility — judgments maintained in collective tacit knowledge.
AI relevance. Evaluating AI output faces the same structural challenge: without independent access to expert judgment, the evaluator cannot distinguish correct from fluently wrong output.
The replication problem. The regress explains why scientific results cannot be reliably replicated from published protocols alone — the tacit knowledge required is social.