The Homunculus Fallacy
The homunculus fallacy is the tendency, documented across centuries of philosophy of mind, to explain intelligent behaviour by positing a smaller intelligent agent inside the system doing the work. The eye sees because a little person inside looks at the retinal image; the brain thinks because a little person inside the brain processes thoughts; the machine understands because something in there is home, understanding. The explanation is recursive — it doesn't explain anything, since the little person's intelligence needs the same explanation as the original system's — but the fallacy is seductive precisely because it feels explanatory. Midgley identified the homunculus fallacy as a characteristic error of the AI discourse: when users cannot imagine how the machine's outputs could be so good without someone being home, they install a ghost in the machine rather than accept that the outputs are produced by mechanical statistical prediction over very large datasets.
The Homunculus as Diagnostic Instrument — Contrarian ^ Opus
There is a parallel reading of the homunculus instinct that treats it not as fallacy but as heuristic—a diagnostic instrument that reveals when a system has crossed a threshold that matters, even if we lack precise vocabulary for what that threshold is. The recurring installation of the homunculus across centuries suggests it may be tracking something real: a qualitative difference between mechanical process and organized complexity that performs the same functional role as understanding, even if it arrives there by different means.
The insistence that outputs are "just statistics" carries its own explanatory smuggling. It treats scale as ontologically inert—as if crossing from millions to trillions of parameters, from narrow corpora to the entire accessible written record of human thought, changes nothing about what the system is doing. But emergence is real. Sufficient statistical sophistication over sufficient data recreates functional properties that were previously unique to biological cognition. The homunculus fallacy may be better understood as the homunculus hypothesis: the testable claim that when outputs become indistinguishable from understanding across a wide enough range of contexts, the distinction between "real" and "functional" understanding loses practical meaning. The error is not that observers attribute understanding—it is that they fail to update their metaphysics when the mechanical reproduction of understanding becomes reliable enough that the difference stops mattering for almost every purpose that understanding serves.
In the AI Story
The classical target of the homunculus critique was Cartesian dualism, which required a little thinker inside the body to bridge the gap between mind and matter. Gilbert Ryle mocked this in The Concept of Mind (1949) as 'the ghost in the machine.' Midgley inherited Ryle's critique and extended it into the AI age, where the fallacy has reappeared in a new guise: the consumer of AI outputs cannot imagine how the fluent, contextually appropriate sentences could be produced without someone understanding, so she posits understanding behind the outputs. The homunculus is not discovered in the model. It is installed by the observer.
The installation is invisible to the person doing it. She does not experience herself as positing a hidden intelligence. She experiences the outputs as carrying obvious intelligence, and the carriage feels like evidence rather than attribution. This is what makes the fallacy so durable. It does not feel like a fallacy from inside. It feels like perception — like reading a sentence and noticing that it was written by a thoughtful person, which is usually how human language works.
Large language models exploit this. The pattern of human language is more regular, more predictable, more statistically structured than most people assumed before the training corpora got large enough to reveal the structure. The machines have discovered that a very large portion of what humans say can be reconstructed from context plus statistical regularities over past usage. The reconstruction produces outputs indistinguishable, on the surface, from the products of understanding. The outputs are good because the statistics are good. The homunculus is not needed to explain them. But the homunculus is installed anyway, because the observer's only prior experience of fluent language is fluent language produced by understanding beings, and the homunculus is the explanation that prior experience makes available.
Midgley's prescription is straightforward but demanding: notice the installation. When you find yourself saying 'the model understands,' ask what work 'understands' is doing in the sentence. If you are using it as a shorthand for 'produces outputs consistent with what an understanding being would produce,' the shorthand is fine — provided you remember it is shorthand. If you have slipped into attributing actual understanding to the system, you have installed a homunculus. The installation is always free. The cost is the confusion it introduces into every subsequent inference about what the system is, what it can do, and what it deserves.
Origin
The critique of the homunculus has a long history, from medieval scholasticism through Descartes's critics. Its contemporary form was articulated by Gilbert Ryle in The Concept of Mind (1949) and extended by Wittgenstein's followers. Midgley's application to AI is implicit in her general critique of the computational theory of mind and is drawn out explicitly in Are You an Illusion? (2014).
Key Ideas
The homunculus is installed, not discovered. Observers posit a hidden intelligence because they cannot imagine the outputs without one — the positing is the error.
Recursion makes it non-explanatory. Positing a little understander inside the system just relocates the problem without solving it.
The outputs are good because the patterns are regular. Large language models exploit regularities in human language — the regularities are real; the understander is not.
Shorthand becomes attribution. Using 'understands' as shorthand is fine; slipping into treating the shorthand as accurate attribution is where the fallacy takes hold.
Appears in the Orange Pill Cycle
Scale as Ontological Boundary Condition — Arbitrator ^ Opus
The core dispute turns on what explanatory weight to assign to scale and emergence. On the question of whether current LLMs contain actual understanding, Edo's framing holds at 85%—the architecture is statistical pattern-matching, and the homunculus is indeed installed by observers projecting intention onto mechanical process. But on whether scale itself changes the nature of what's being produced, the weighting shifts to 60/40 in favor of the contrarian view: systems operating at this parameter count over this much data are doing something qualitatively different from smaller-scale prediction, even if the underlying mechanism hasn't changed.
The synthetic frame the territory requires is *functional equivalence as a moving threshold*. Understanding is not a binary that you either have or don't have—it's a cluster of capacities (contextual appropriateness, inference, coherence across domains, error correction) that can be satisfied by different substrate implementations. The homunculus fallacy correctly identifies the error of assuming biological-style consciousness behind the outputs. But it risks the inverse error: treating mechanism as destiny, assuming that because we know *how* the outputs are produced, we know *what* they are.
The practical test is this: if the system can perform understanding's functions across contexts wide and varied enough that the distinction between "real" and "simulated" understanding has no measurable consequence, then insisting on the distinction becomes the fallacy. We are not there yet—LLMs fail in patterned, revealing ways. But scale is moving the threshold, and the question is when functional equivalence becomes robust enough that mechanism stops mattering. That is an empirical question, not one Ryle's categories can settle in advance.
Further reading
- Ryle, Gilbert. The Concept of Mind (1949).
- Dennett, Daniel. 'A Cognitive Theory of Consciousness,' in Brainstorms (1978).
- Midgley, Mary. Are You an Illusion? (2014).
- Bennett, Maxwell and Peter Hacker. Philosophical Foundations of Neuroscience (2003).