The Whole Machine

EDO SEGAL: John, intellectual honesty made you do something rare in your 1980 paper — you stated the strongest objection to your own argument before your critics could, and then answered it. It is the most important counter to the Chinese Room, and the one many of your sharpest opponents think you never cleanly beat. The Systems Reply. State it fairly, then give your answer, and then Alan gets to push.

SEARLE: I will state it at full strength, because a weak version would be cheating. The reply says: of course the man does not understand Chinese — but the man is just the central processor. The whole system — man plus rulebook plus the room plus the database — understands, and you are pointing at one component and concluding the system has no property the component lacks. That is a fallacy, like saying a man cannot be alive because his individual neurons are not. So the understanding belongs to the system, not the man. Now my answer, which is to internalise the system. Let the man memorise the rulebook, hold the entire database in his head, throw away the room, and work in an open field. He is now the whole system, with nothing left over, and he still understands no Chinese. There is no system above and beyond the man for the understanding to hide in. That is the hinge the whole Chinese Room turns on. I will confess I once wrote that I found it embarrassing to have to answer this objection at all — and I have come to think that flash of impatience was a tell. It is not a silly objection. But internalising the system is the right answer, and it holds.

TURING: It is a tell, and I am going to press exactly on it, warmly, because this is where I think your room cracks. When the man memorises the rulebook and runs it in his head, you ask him whether he understands Chinese, he introspects, he finds nothing, and you take his report as decisive. But introspection is the single faculty we have most reason to distrust about whether understanding is happening at a level the narrator cannot survey. Your visual cortex is solving staggering geometric problems right now that you have no inner access to whatsoever. The man's English-speaking self reports no Chinese — granted. But the critic's claim is that he is now implementing a second process, a Chinese-understanding system running on the same wetware that runs his English self, and that this implemented system might understand Chinese while the host sincerely reports it does not. You cannot refute that by having the host introspect, because the host is precisely the part the critic says is out of the loop. Your whole verdict rests on the intuition that "if understanding were happening in me, I would feel it" — and that intuition is the very thing in dispute.

SEARLE: And here is why the intuition is not the cheat you want it to be. We are not talking about subpersonal vision, where I never claimed to introspect the geometry. We are talking about understanding a language — a paradigmatically conscious achievement. When I understand English, there is something it is like for me; the meanings are present to me. The claim that there is a second self in my skull, fluently understanding Chinese, having Chinese meanings present to it, entirely sealed off from the only self that is conscious — that is not a modest hypothesis. It is the postulation of a whole second conscious subject on no evidence except that the behaviour came out right. You are multiplying minds to save the theory. The simpler reading is the one in front of us: symbol manipulation produced correct output and no understanding, because that is what symbol manipulation does.

TURING: Or the simpler reading is that understanding never was the felt glow you are insisting on, and that you are denying it to the system for the same reason a man denies the cortex its geometry — because the doing of it is not delivered to the narrator as a feeling. You keep saying "there is something it is like," and I keep agreeing there is, for you, and asking how you know there is not, for the process running the Chinese. The honest answer is that you do not know. You have an intuition pump — and it pumps, beautifully, by making me picture a slow man with paper, so that the system's possible understanding is unimaginable simply because I cannot visualise a billion operations a second distributed across a field. Change the tempo in imagination and the intuition wobbles. That does not prove me right. It proves the room persuades partly by stagecraft.

SEARLE: I will grant the stagecraft and still keep the verdict, and that is not a dodge — it is the whole difference between us. Every intuition pump can be accused of theatre; the question is whether, when you strip the theatre, the point survives. Strip mine and it survives as a syllogism with no pictures at all. Programs are syntactic. Minds have semantics. Syntax is not sufficient for semantics. Therefore programs are not sufficient for minds. You can attack any premise you like. You cannot attack the staging, because the staging was only ever a way to make the premises vivid. And notice what you keep doing, Alan — every time I corner the argument, you escape by deflating the human side: understanding was never felt, reference was never grounded, the narrator was always confabulating. Your defence of the machine has quietly become a demotion of us. I notice your epistemics loosen wonderfully whenever the conclusion would be interesting.

TURING: And yours tighten wonderfully whenever the conclusion would be uncomfortable. We have each named the other's failure mode. That may be the truest exchange we have all night.

EDO SEGAL: I want to name what just happened, because the reader cannot see the room: that was the first exchange where neither of you was performing for me — you were arguing for keeps. And notice the strange topology. Alan needs the man-in-the-field to maybe-understand without knowing it; John needs introspection to be trustworthy exactly where Alan needs it to fail. The whole quarrel has narrowed to a single question about self-knowledge. Hold that, because it detonates in the consciousness round. Next, the thing both of you have been circling — how the machine got its competence in the first place. Not programmed. Raised. The child machine, and the know-how no rule can hold. After this.