The Demarcation Problem
Popper's original demarcation problem asked how to draw the line between genuine science and pseudoscience. His answer was falsifiability: a theory is scientific if it specifies the conditions under which it would be wrong. Eighty years later, the problem has found its most consequential new application. The question is no longer how to distinguish science from pseudoscience but how to distinguish genuine insight from plausible fabrication in the output of AI systems that produce both with identical surface quality. The traditional markers professionals have used for generations to evaluate reliability — prose quality, logical structure, appropriate citation, confident tone — have been decoupled from the quality they were supposed to indicate. Good prose no longer requires understanding. The heuristics are broken, and the replacement heuristics have not yet been widely developed.
In the AI Story
The original problem was empirical: Popper looked at Marxism, Freudianism, and Adlerian psychology and asked what distinguished them from physics. The answer was not their subject matter but their structure. Einstein's theory put itself at risk through specific predictions that could fail. The psychoanalytic theories did not. They absorbed all evidence. The demarcation line ran between risk and immunization.
The AI version of the problem is structurally analogous but operationally new. A lawyer receives a draft brief. A researcher receives a literature review. A student receives an essay. Each is fluent, coherent, confidently presented. Each may be accurate, partially accurate, or substantially fabricated. The reader cannot tell from the surface. The heuristics that used to correlate reliability with presentation — good prose meant genuine understanding, because producing good prose required understanding — no longer apply. The correlation has been severed. Good prose is now produced by a system that does not understand.
The deeper problem, which this book highlights in the Deleuze failure, is not fabricated facts but fabricated insight. A fact can be checked against a database. An insight — a claim about how ideas relate — requires the kind of deep engagement with the subject matter that the AI tool was supposed to save the user from. The verification requires the very work the tool was designed to replace. This creates the circularity that makes the demarcation problem in the AI age uniquely difficult: the tool's utility and the user's evaluative capacity exist in inverse proportion.
Solving the demarcation problem in the AI age requires new criteria analogous to Popper's falsifiability. Every significant AI output should flag the boundaries of its confidence, specify what would refute it, and distinguish between what the system is confident about and what it is extrapolating. Current architectures do none of this by default. The discrimination must be supplied by the user, who must know the costume is a costume.
Origin
Popper introduced the demarcation problem in Logik der Forschung (1934) and returned to it across his career, most fully in Conjectures and Refutations (1963). The AI application is recent, emerging in work by Donald Gillies, Ming Li, and researchers at Stanford (the 2025 POPPER framework). Ming Li's 2024 paper argues explicitly that the AI field itself suffers from a demarcation problem — many claims about AI capability are presented without specifying refutation conditions, making them pseudoscientific by Popper's criterion.
Key Ideas
Structural diagnosis. The demarcation problem is about how claims are structured, not what they are about. A claim immune to refutation is pseudoscientific regardless of its subject.
Decoupled markers. AI has severed the historical correlation between presentation quality and reliability, breaking the heuristics professionals depended on.
Fabricated insight. The hardest AI failures are not wrong facts but wrong relationships between ideas, which cannot be verified against databases.
Verification circularity. Checking AI output requires the expertise the tool was supposed to replace, creating a structural paradox in evaluation.
New criteria needed. The AI age demands demarcation criteria analogous to falsifiability: self-disclosed confidence bounds, specified refutation conditions, flagged extrapolation.
Appears in the Orange Pill Cycle
Further reading
- Popper, Karl. Conjectures and Refutations. Routledge, 1963.
- Li, Ming. "The Boundaries of AI: A Popperian Approach." University of Waterloo, 2024.
- Huang, Kexin et al. "POPPER: Automated Falsification of Scientific Hypotheses." Stanford, 2025.
- Lakatos, Imre. The Methodology of Scientific Research Programmes. Cambridge University Press, 1978.