Verification Circularity

In The You On AI Field Guide

Fabricated facts can be checked externally — against databases, cited sources, empirical records. Fabricated insight cannot. An insight is a claim about how ideas relate, and evaluating such a claim requires understanding the ideas at a depth sufficient to judge the relationship. No database can supply this. The verification has to come from the user's own prior engagement with the subject matter — the engagement that using the tool circumvents.

The circularity is not resolvable at the tool level. Improvements to model accuracy reduce the rate of fabrication but do not eliminate it, and cannot eliminate it — statistical generation will always produce some plausible continuations that are wrong. As the rate declines, users trust the output more, which means they verify less, which means the errors that do appear are more likely to propagate unchecked. Higher accuracy does not solve the problem. It changes its character.

The resolution must come from outside the tool. It requires practices, institutions, and habits that maintain the user's capacity for critical evaluation independent of the tool — what Segal calls dams and what Popper would call the institutional structures of critical rationalism. Protected time for direct engagement with source material. Deliberate practice without the tool, so evaluative capacity is maintained. Institutional norms that require verification even when it feels inefficient.

The circularity's most troubling implication is generational. If the current generation of professionals developed their evaluative capacity through direct engagement with sources before AI tools arrived, they retain the capacity to verify — though they may use it less as the tool becomes more convenient. The next generation, trained from the start on AI-assisted workflows, may not develop the capacity in the first place. What atrophies in the current generation is pre-formed; what fails to develop in the next may never be built. This is the deepest challenge verification circularity poses to the long-term health of expert communities.

Origin

The concept is developed in Chapter 4 of this volume as a structural feature of the AI demarcation problem. It generalizes observations made in You On AI about the Deleuze failure and the difficulty of catching fabricated insight.

Key Ideas

Inverse proportion. The tool's utility and the user's evaluative capacity pull in opposite directions.

Insight vs. fact. The circularity bites hardest at the level of fabricated insight, which has no external check.

Unresolvable at the tool level. Accuracy improvements change the problem's character but do not eliminate it.

External resolution required. The solution lies in practices and institutions that maintain critical capacity independent of the tool.

Generational risk. Current professionals inherited capacity that may not develop in those trained from the start on AI-assisted workflows.