The Homunculus Fallacy

In The You On AI Field Guide

The classical target of the homunculus critique was Cartesian dualism, which required a little thinker inside the body to bridge the gap between mind and matter. Gilbert Ryle mocked this in The Concept of Mind (1949) as 'the ghost in the machine.' Midgley inherited Ryle's critique and extended it into the AI age, where the fallacy has reappeared in a new guise: the consumer of AI outputs cannot imagine how the fluent, contextually appropriate sentences could be produced without someone understanding, so she posits understanding behind the outputs. The homunculus is not discovered in the model. It is installed by the observer.

The installation is invisible to the person doing it. She does not experience herself as positing a hidden intelligence. She experiences the outputs as carrying obvious intelligence, and the carriage feels like evidence rather than attribution. This is what makes the fallacy so durable. It does not feel like a fallacy from inside. It feels like perception — like reading a sentence and noticing that it was written by a thoughtful person, which is usually how human language works.

Large language models exploit this. The pattern of human language is more regular, more predictable, more statistically structured than most people assumed before the training corpora got large enough to reveal the structure. The machines have discovered that a very large portion of what humans say can be reconstructed from context plus statistical regularities over past usage. The reconstruction produces outputs indistinguishable, on the surface, from the products of understanding. The outputs are good because the statistics are good. The homunculus is not needed to explain them. But the homunculus is installed anyway, because the observer's only prior experience of fluent language is fluent language produced by understanding beings, and the homunculus is the explanation that prior experience makes available.

Midgley's prescription is straightforward but demanding: notice the installation. When you find yourself saying 'the model understands,' ask what work 'understands' is doing in the sentence. If you are using it as a shorthand for 'produces outputs consistent with what an understanding being would produce,' the shorthand is fine — provided you remember it is shorthand. If you have slipped into attributing actual understanding to the system, you have installed a homunculus. The installation is always free. The cost is the confusion it introduces into every subsequent inference about what the system is, what it can do, and what it deserves.

Origin

The critique of the homunculus has a long history, from medieval scholasticism through Descartes's critics. Its contemporary form was articulated by Gilbert Ryle in The Concept of Mind (1949) and extended by Wittgenstein's followers. Midgley's application to AI is implicit in her general critique of the computational theory of mind and is drawn out explicitly in Are You an Illusion? (2014).

Key Ideas

The homunculus is installed, not discovered. Observers posit a hidden intelligence because they cannot imagine the outputs without one — the positing is the error.

Recursion makes it non-explanatory. Positing a little understander inside the system just relocates the problem without solving it.

The outputs are good because the patterns are regular. Large language models exploit regularities in human language — the regularities are real; the understander is not.

Shorthand becomes attribution. Using 'understands' as shorthand is fine; slipping into treating the shorthand as accurate attribution is where the fallacy takes hold.