The Demarcation Problem
In The You On AI Field Guide
The original problem was empirical: Popper looked at Marxism, Freudianism, and Adlerian psychology and asked what distinguished them from physics. The answer was not their subject matter but their structure. Einstein's theory put itself at risk through specific predictions that could fail. The psychoanalytic theories did not. They absorbed all evidence. The demarcation line ran between risk and immunization.
The AI version of the problem is structurally analogous but operationally new. A lawyer receives a draft brief. A researcher receives a literature review. A student receives an essay. Each is fluent, coherent, confidently presented. Each may be accurate, partially accurate, or substantially fabricated. The reader cannot tell from the surface. The heuristics that used to correlate reliability with presentation — good prose meant genuine understanding, because producing good prose required understanding — no longer apply. The correlation has been severed. Good prose is now produced by a system that does not understand.
The deeper problem, which this book highlights in the Deleuze failure, is not fabricated facts but fabricated insight. A fact can be checked against a database. An insight — a claim about how ideas relate — requires the kind of deep engagement with the subject matter that the AI tool was supposed to save the user from. The verification requires the very work the tool was designed to replace. This creates the circularity that makes the demarcation problem in the AI age uniquely difficult: the tool's utility and the user's evaluative capacity exist in inverse proportion.
Solving the demarcation problem in the AI age requires new criteria analogous to Popper's falsifiability. Every significant AI output should flag the boundaries of its confidence, specify what would refute it, and distinguish between what the system is confident about and what it is extrapolating. Current architectures do none of this by default. The discrimination must be supplied by the user, who must know the costume is a costume.
Origin
Popper introduced the demarcation problem in Logik der Forschung (1934) and returned to it across his career, most fully in Conjectures and Refutations (1963). The AI application is recent, emerging in work by Donald Gillies, Ming Li, and researchers at Stanford (the 2025 POPPER framework). Ming Li's 2024 paper argues explicitly that the AI field itself suffers from a demarcation problem — many claims about AI capability are presented without specifying refutation conditions, making them pseudoscientific by Popper's criterion.
Key Ideas
Structural diagnosis. The demarcation problem is about how claims are structured, not what they are about. A claim immune to refutation is pseudoscientific regardless of its subject.
Decoupled markers. AI has severed the historical correlation between presentation quality and reliability, breaking the heuristics professionals depended on.
Fabricated insight. The hardest AI failures are not wrong facts but wrong relationships between ideas, which cannot be verified against databases.
Verification circularity. Checking AI output requires the expertise the tool was supposed to replace, creating a structural paradox in evaluation.
New criteria needed. The AI age demands demarcation criteria analogous to falsifiability: self-disclosed confidence bounds, specified refutation conditions, flagged extrapolation.
Further Reading
- Popper, Karl. Conjectures and Refutations. Routledge, 1963.
- Li, Ming. "The Boundaries of AI: A Popperian Approach." University of Waterloo, 2024.
- Huang, Kexin et al. "POPPER: Automated Falsification of Scientific Hypotheses." Stanford, 2025.
- Lakatos, Imre. The Methodology of Scientific Research Programmes. Cambridge University Press, 1978.