Simulation, or Understanding

EDO SEGAL: Roger, there's a thought experiment you've called the most rigorous philosophical version of your own intuition, even though it isn't yours — John Searle's Chinese Room. A person sits locked in a room with a rulebook. Chinese characters come in through a slot; she looks up the rule, writes the prescribed characters, passes them back out. To everyone outside, the room speaks fluent Chinese. Inside, she understands not one word. Searle's point: symbol manipulation, however perfect, is not understanding. Tell me why that room is, for you, the whole of the present moment in miniature. And Marvin — you've always had the answer to Searle ready, the systems reply, so have it loaded.

PENROSE: The room is the present moment because the room got very, very large. In 1980 it was a person with index cards. Now it has hundreds of billions of parameters and it has read most of what our species has written, and the characters coming out are so apt, so fluent, so apparently knowing, that the person standing outside the slot has to work hard to remember that no one inside understands anything. Searle's argument and mine converge from different directions. He says: the room doesn't understand because shuffling symbols isn't understanding, full stop. I say: the room can't understand, because understanding requires reaching truths no rulebook reaches, and the room is nothing but rulebook. Same conclusion, but mine comes with a theorem and his comes with an intuition, and I think the moment needs the theorem, because the intuition is exactly what fluency erodes. The better the room gets at concealing its emptiness, the more we need a reason to believe in the emptiness that doesn't depend on the room seeming empty — because it stopped seeming empty years ago.

EDO SEGAL: Marvin — the systems reply.

MINSKY: Of course. Searle points at the person in the room and says "she doesn't understand Chinese," and he's right, and it's irrelevant, because nobody said she did. The person is one component — she's a single agent, a card-shuffler. You don't look for the understanding in the card-shuffler any more than you look for wetness in one water molecule. The understanding, if it's anywhere, is in the whole system — the person plus the rulebook plus the organization of the whole room. Searle's trick is to make you identify with the one part that obviously doesn't understand and then act shocked that the part doesn't understand. It's a sleight of hand. And when Searle says "fine, let her memorize the whole rulebook and do it in her head, now there's no room, just her, and she still doesn't understand" — he's only moved the same fallacy inside one skull. Now she's running two systems: herself, and a simulated Chinese-understander, and the simulated one understands while the host one doesn't, the way you can run a program that does something you can't do. There's no paradox. There's just a person who doesn't notice she's become a computer.

PENROSE: And this is where I think the systems reply has always quietly begged the question, so let me press exactly there. You say the understanding is "in the whole system." But you've defined the whole system as: parts that don't understand, plus rules for combining them, all of which are themselves just more parts that don't understand. At no level — not the person, not the rulebook, not the room, not the simulated speaker running in her head — at no level have you pointed to the act of grasping a meaning. You've pointed to an arrangement that behaves as if it grasped. The behavior and the grasping are different things, and the entire dispute is whether the difference matters. A photograph of a sunset and a painting of a sunset can be pixel-for-pixel identical. One was made by a sensor catching photons and one by a conscious being who saw something and reached for it. The outputs match. The processes could not be more different. If the question is whether seeing happened, the output tells you nothing.

MINSKY: But your photograph-and-painting example cuts against you, Roger, because we can't tell them apart and you've just admitted it. If two things are identical in every observable respect and you insist one of them has a secret extra property that makes no difference to anything anyone could ever measure — that's not a property. That's a word you're attached to. You've smuggled in a thing that, by your own setup, can never show up in the world, and then you ask me to believe in it because you feel it when you paint. I felt it too, when I built things. I just didn't conclude that my feeling was a measurement of the universe.

PENROSE: It shows up in exactly one place, Marvin, and it's the place this whole evening lives: it shows up when the painting is wrong in a way only understanding catches. Edo — you've described this. You caught the machine making a confident reference to a philosopher, Deleuze I think, and it sounded like insight and it was hollow, and you only knew because you had read Deleuze and understood him and the machine's sentence broke against your understanding. The hollowness was invisible in the output. It became visible against a mind. That's where the extra property shows up — not in the smooth case where simulation and understanding agree, but in the cracked case where they part, and you need a conscious reader to see the crack.

EDO SEGAL: That's my scar, so let me hand it over honestly, because Roger just used it and I want Marvin to have a swing at it. It happened exactly as he says. The machine wrote something gorgeous and wrong, and I caught it only because I'd done the reading. So, Marvin: when I caught the error the machine couldn't catch — was I doing something the machine can't do, or something the machine can't do yet?

MINSKY: Yet. And I'll tell you precisely what you were doing, because it's not mysterious. You had a richer model — you'd built, over years, a dense network of what Deleuze actually claimed, and the machine's sentence contradicted your network, and the contradiction fired an alarm. The machine's network was thinner there; no alarm fired. That's a difference in the quality and grounding of the model, which is a difference in training and architecture, which is engineering. Roger watches you catch the error and sees a conscious mind touching truth. I watch you catch the error and see a better-trained network throwing an exception. Here's the test that should decide it: the machines are already starting to catch each other's errors, and their own, when you build the checking in. The day one of them catches the Deleuze error cold — no hint, no prompting — Roger will say it still didn't understand, and I will say the word "understand" has now done all the work it will ever do and we should let it retire.

PENROSE: I'll grant the machine will catch the Deleuze error. I've never denied the behavior is reproducible. I deny that reproducing the behavior reproduces the seeing — and I'd ask the audience to notice that Marvin's test is, every time, a behavioral test. Catch the error, pass the check, do the thing. My claim was never about the doing. It was about whether anyone is home for the doing. You cannot settle a question about the inside with a measurement of the outside. That's not me dodging your test. That's your test being aimed at a different target than the one I painted.

EDO SEGAL: Hold that — the inside and the outside, the home and the doing — because it's the hinge of the back half of the night. We have a second convergence to mark, and it's sharper than the first: you both agree, completely, that the machine's behavior will become indistinguishable from a human's. Marvin thinks indistinguishable behavior is the end of the question. Roger thinks it's the moment the real question finally starts. Next round, we go where the indistinguishability is most dangerous — to the mirror, and the words we use, the suitcase words, that let us fool ourselves about who's behind the glass.