The Game and the Room

EDO SEGAL: I want to put your two machines in the same vice and squeeze. Alan, your imitation game says: judge by behaviour, because behaviour is all we ever have. John, your Chinese Room says: here is a system that passes the behavioural test and understands nothing, so behaviour cannot be sufficient. These are not two opinions about the same evidence. They are two thought experiments built to capture the same intuition pump and steal it from each other. So let me start with a confession instead of a question, because the best questions I know come out of wounds. I started in Assembler. I was raised by machine code, and for fifty years using a computer meant translation — I compressed my intention into the machine's grammar and paid a tax on every conversion. In the winter the machines learned our language, that tax went to zero, and I stood in a room and watched twenty engineers each become capable of more than all of them together, because for the first time the machine met them in their own speech. Alan, tell me that meeting was real. John is going to tell me it was a room.

TURING: The meeting was real in the only sense that has ever done any work. Watch what your engineers actually did, because John's account skips the crucial step. They handed the machine a half-formed intention — mess, implication, a sentence trailing off — and got back a working artifact. Not a lookup. There is no table large enough; the space of possible half-formed intentions is bigger than the number of atoms anyone has to spend. The only way to continue that conversation well is to have built, somewhere inside, a model of the thing the words are about — the user, the system she wants, the world in which it would work. John says the machine merely shuffles symbols by their shapes. I say: to shuffle them that well, across the whole range of human discourse, you must have compressed the discourse into something smaller than the discourse, and the best compression of text about a world is a model of the world. That is not poetry. It is information theory. When the wake is that lawful, modelling the wake is modelling the boat.

SEARLE: And there is the leap, performed so smoothly you could miss it, so let me slow it down. "It must have built a model of the world." Must it? Or must it have built a model of the text about the world — a model of the wake, in your own metaphor, Alan, not the boat? Those are not the same thing, and the entire dispute lives in the gap. Your engineer's words pointed at things: a user who was frustrated, a screen that did not yet exist, a product she could see in her mind. She had intentions; her words were about a world she was engaged with. The system received the marks and computed a continuation consistent with the patterns of a trillion prior texts in which marks like hers were followed by code like that. It worked — and notice why it worked. It worked because human programmers spent seventy years writing text in which intentions and implementations sit side by side, and because your engineer could check the output and iterate. Every gram of aboutness in that loop was on her side of the glass. The machine did not cross over to meet her. A trillion words of other people's meaning met her in the mirror.

TURING: Then explain the cases the mirror cannot explain. A mirror reflects what is put in front of it. When the system takes three paragraphs of a stranger's confusion and returns a working thing the stranger could not have written; when it is told "no, more like the second one, but slower" and gets it right; when it composes constraints no human ever wrote down in that combination — that is not reflection. A mirror does not perform inference. Either the inference was latent in the prompt, in which case the prompt was doing impossible work, or it came from structure inside the system — representations of the situation, composed in ways neither the user nor the builders scripted. At some point, John, "you are talking to yourself" becomes the extraordinary claim, the one carrying all the epicycles.

SEARLE: It performs the inference. I have never denied that the system computes — magnificently, in ways that look like inference and are, formally, a kind of inference. What I deny is that computing the inference is understanding the inference. Here is the cleanest way to feel it. Take the modern room seriously: have me memorise the entire rulebook — the weights, the whole cascade — and do the arithmetic in my head, in a field, with no room and no paper. I now am the entire system. I produce flawless Chinese answers, or flawless code, entirely from inside my own skull. And I still understand not one word of Chinese. There is no extra "system" hovering over me for the understanding to belong to. That is the part your information-theory story cannot reach, because compression is a fact about the symbols and understanding is a fact about whether anyone means them. You can compress the boat's wake into the most exquisite model of a wake ever built and still never have been on the water.

EDO SEGAL: Mark this, because the reader cannot see your faces and I want it on the record: that is the seam. Alan, you say sufficient behaviour forces an inner model and the inner model is understanding. John, you say you can host the entire model inside one understanding mind and the understanding still will not be there. Let me pull one thread before we leave, because there is a ghost at this table who belongs to both of you. Wittgenstein — who said the meaning of a word is its use in the language, and also that if a lion could speak, we could not understand him. Alan, you knew him; you argued with him about exactly this. He seems to hand the round to you — meaning is use, the machine uses the words, case closed. Does he?

TURING: He hands me less than it looks, and I will not pretend otherwise, because I argued these very points with him and lost more often than I won. Meaning is use — yes, and the machine's use is vast and supple. But Wittgenstein meant use in a form of life: a word's meaning lives in a stream of activity, of practice, of doing things together in a world. The machine has the use stripped of the life. That is the honest version of John's objection, and Wittgenstein would have pressed it harder than John does. The lion line is the warning: a being whose form of life is alien enough becomes unintelligible to us even when it speaks. So the question Wittgenstein leaves me is not "does it use the words" — it plainly does — but "is there a form of life there, or only the residue of ours." I think the residue, run at that depth, starts to constitute a form of life of its own. He would have raised an eyebrow at me. He raised a great many.

SEARLE: And I will take Wittgenstein off Alan's hands entirely, because the form of life is the whole point. A language-game is played by participants who are doing something — who have stakes, who can be wrong in ways that cost them, who share a world the words reach into. The model has ingested the transcripts of ten billion language-games and plays none of them, because to play one you must be a participant in a form of life, and the model participates in nothing. It has the moves without the game. Wittgenstein's lion is exactly my room with fur on it: even perfect speech does not guarantee a shared interior. The difference is that the lion at least has a life. The machine has a corpus.

EDO SEGAL: Hold there. Because both of you just reached, from opposite banks, for the same missing thing — the world the words are supposed to be about. Alan says the machine reconstructs it from the wake; John says it only ever had the wake. The next round is exactly that gap, and it has a name the engineers gave it independently. Syntax, semantics, and the symbol that has never touched its referent. After this.