The Substitution Fallacy
The substitution fallacy is Flyvbjerg's name for the dominant cognitive error in contemporary AI discourse: the assumption that functional indistinguishability between machine output and human output entails substantive equivalence between machine capability and human capability. The AI system that writes code, drafts legal briefs, or generates medical diagnoses is performing operations previously performed by human experts. The output may be functionally identical to what the expert produced. But the process by which it was produced is fundamentally different — pattern-matching across training data versus embodied knowledge built through years of practice — and the difference matters because the developmental process, not the output it produces, is the substrate on which phronesis is built.
The Developmental Process as Luxury Good — Contrarian ^ Opus
There is a parallel reading that begins not with the expert's trajectory but with the material conditions under which expertise has historically been available. The substitution fallacy presumes that the developmental process Flyvbjerg identifies—years of embodied practice, formative struggle, apprenticeship under masters—was widely accessible rather than a privilege of specific class positions, institutional settings, and historical moments. For most people performing most work throughout history, expertise development was constrained by time scarcity, economic necessity, and structural exclusion. The plumber who cannot afford four hours to complete a job that AI reduces to ten minutes is not experiencing phronetic erosion—they are experiencing the first opportunity to remain economically viable.
The critique of output-focused substitution assumes that the developmental process was the primary value being produced, but from the perspective of the client paying for plumbing, the patient needing a diagnosis, or the startup needing code, the output was always the thing being purchased. The phronetic capacity of the expert was instrumentally valuable insofar as it reliably produced the output; if the output can be produced reliably without it, the capacity's absence is a cost reduction, not an erosion. This does not make Flyvbjerg wrong about what is lost—it makes him right about something that matters less than he assumes to the vast majority of contexts where substitution occurs. The developmental process as the site of human flourishing is a perspective available primarily to those whose material conditions already permit the luxury of development untethered from immediate output demands.
In the AI Story
The fallacy operates at two levels. At the individual level, it treats the production of expert-like output as evidence that the machine has replicated expert knowledge — ignoring the fact that expert knowledge includes capacities the machine does not possess: the ability to recognize when rules fail, the judgment to weigh incommensurable values, the stake in outcomes that grounds responsibility. At the institutional level, the fallacy treats the automation of expert output as equivalent to the automation of the function expertise serves — ignoring the fact that the function depends on the developmental process, not just the output.
The ten minutes case Segal describes — the engineer whose routine plumbing contained the rare moments of formative struggle — illustrates the operational consequence. When AI automates the plumbing, it removes the tedium and the formative moments together, because from the system's perspective the entire four hours is plumbing, defined functionally rather than developmentally. The substitution appears successful because the output is maintained. The phronetic erosion it produces is invisible to any measurement that evaluates substitution by output rather than by the developmental trajectory of the person producing it.
The fallacy has an institutional analog in the commoditization of code that produced the 2026 Software Death Cross. Organizations whose value propositions rested on the scarcity of technical capability discovered that the capability had been commoditized and their valuations had to be repriced. But the substitution was real only at the techne layer. The phronetic layer — institutional knowledge, customer relationships, workflow patterns, regulatory expertise, cultural understanding — remained. The companies that mistook the techne commoditization for the substitution of their entire value proposition learned the lesson painfully; those that recognized the distinction survived.
The corrective is methodological vigilance. Every claim that AI has replaced a category of human work must be examined for the substitution fallacy: has the output been replicated, or has the capacity been replaced? These are different questions, and conflating them produces the systematic overestimation of AI substitution that now pervades both corporate strategy and public discourse. The honest answer in most cases is that the output has been replicated while the capacity has been partially eroded and partially preserved in ways that vary by context and that only phronetic analysis can detect.
Origin
The concept is implicit throughout Flyvbjerg's phronesis work and is made explicit in his 2025 AI writings. Related arguments appear in Hubert Dreyfus's long engagement with AI and in Shannon Vallor's work on moral deskilling.
Key Ideas
Output versus process. Functional equivalence of output does not entail substantive equivalence of capability — the developmental process is a separate dimension.
Invisible erosion. When substitution replaces the developmental process, the phronetic erosion is invisible to measurement that evaluates only output.
Institutional consequence. Organizations that mistake output commoditization for capability substitution misprice their own value and their competitors'.
Generational compounding. Each generation of practitioners trained through substituted processes loses the capacity the process developed, producing cumulative erosion invisible within any single generation.
Corrective vigilance. Every substitution claim must be examined separately for output replacement versus capacity replacement, because the two diverge systematically.
Appears in the Orange Pill Cycle
Domain-Variant Substitution Mapping — Arbitrator ^ Opus
The right framing depends entirely on which domain and which question you're asking. For safety-critical decisions where edge cases dominate—medicine, structural engineering, crisis response—Flyvbjerg's distinction is approximately 90% correct: the output may be statistically adequate but the absence of phronetic capacity to recognize when the pattern fails is catastrophic, and the erosion of developmental pathways ensures the capacity cannot be rebuilt when needed. For high-volume, low-stakes outputs where consistency matters more than edge-case judgment—basic document drafting, routine code, standard diagnostics—the contrarian view is roughly 70% right: the output was always the thing being purchased, and AI substitution expands access while reducing cost in ways that benefit more people than the phronetic loss harms.
The synthesis requires acknowledging that expertise development was never universally accessible, so its erosion is not uniformly distributed loss—it concentrates among those who had access to developmental pathways, while substitution benefits concentrate among those who had access only to expensive expert outputs. The developmental process matters most precisely where Flyvbjerg claims: at the formative margins where struggle builds capacity. But those margins were always available primarily to people whose economic position permitted extended development, and AI substitution does not erase them—it shifts who can afford them.
The corrective is not choosing between the views but mapping substitution domain by domain: where does phronetic capacity matter enough that its erosion outweighs access expansion (medicine, law, engineering design), and where does output adequacy matter enough that development cost outweighs capacity preservation (routine drafting, standard diagnostics, basic code)? The answer varies, and the variation is the terrain on which substitution policy must be built.
Further reading
- Dreyfus, Hubert. What Computers Still Can't Do. MIT Press, 1992.
- Vallor, Shannon. Technology and the Virtues. Oxford University Press, 2016.
- Flyvbjerg, Bent. 'AI as Artificial Ignorance.' Project Leadership and Society, 2025.
- Ericsson, K. Anders. Peak: Secrets from the New Science of Expertise. Houghton Mifflin Harcourt, 2016.