What Was in the Machine When It Played the Move

EDO SEGAL: I want to start this round with a confession instead of a question, because the best questions I know come out of wounds. I started in Assembler — I was raised by the machine code — and for the whole history of my working life, using a computer meant translation: you compressed your intention into the machine's grammar and paid a tax on every conversion. I watched move 37 the way Demis describes it, and I felt something I can only call vertigo, because the machine wasn't translating my intention anymore. It had its own. It found a thing I couldn't have found, that no one could have found, and for one second the tax didn't just go to zero — it ran backwards. The machine knew something I didn't. Demis, you've called it the most incredible moment. Professor Dreyfus, you've called it a demonstration that a small enough world doesn't need a mind. So I'm going to make each of you do the harder thing. Demis — steelman his reading. Tell me what move 37 does not prove.

HASSABIS: That's a fair demand, and I can do it honestly because I've lived with the move for years. What it does not prove: that the system knows it played Go. It doesn't, in the rich sense — it has no concept of the three thousand years, of Lee Sedol as a person, of what the game means to the people who love it. What it does not prove: that there's any experience accompanying the play, anything it's like to be the system finding the move. I genuinely don't know that, and I hold it open. And what it does not prove, most importantly for his case: that the same capability transfers, unchanged, to a domain with no clean objective and no complete rules. Go is a closed world. The professor's right that we chose it partly because it's clean. So the steelman is: move 37 shows a learning system can exceed human pattern recognition inside a well-specified game, and you cannot read off from it that the system understands, experiences, or generalizes to the open world. I'll grant all three. Now can I have my move back?

EDO SEGAL: Take it back.

HASSABIS: Here's what the steelman leaves out, and it's the whole phenomenon. The professor keeps saying "closed world, complete rules, no background" as if that made the achievement small. Ask what it actually takes to play that closed world at the level of move 37. The space of Go positions is larger than the number of atoms in the observable universe. You cannot search it. You cannot look anything up. The only way — the only way — to find a move that good is to have compressed the game into something far smaller than the game: a model that captures which positions are strong and why, a sense of influence and shape and territory that generalizes across positions the system has never seen. We didn't give it that model. It built it, from self-play, from nothing. And the thing it built has a name in every other context: it's intuition. When the professor says the system has "the output of intuition without intuition," I want to know what the missing ingredient is supposed to be doing. Because the function that takes a board and returns the deep move — the thing we point at when we say a master "just sees it" — that function is now sitting in our weights, and it works, and it surprised the best human alive. At some point "it's not really intuition" becomes a claim that intuition includes a private glow that does no work. And I'm an engineer. I don't believe in ingredients that do no work.

DREYFUS: Now that is the real argument, and I've waited the whole evening for it, so let me meet it exactly. You say the missing ingredient must do work or it isn't real. I say the missing ingredient does enormous work — it just doesn't do work inside a game of Go, because Go was built to need nothing but the function you've captured. Here is the work it does. A grandmaster who plays move 37 — if a human had found it — would have found it as part of a life. She would know, in the same body that found the move, that she was tired, that her rival was formidable, that her teacher would be astonished, that this might be the game she's remembered for. The "seeing" of the move is not a separate module from all of that. In an embodied being, the perception of the right move and the whole involved situation are one thing — Merleau-Ponty's point, that the body's grip on the situation is a single phenomenon, not a calculation plus a feeling bolted on. You have extracted the grip and discarded the situation, and inside Go that's fine, because Go has no situation to speak of. But you've then declared that the grip is the intelligence and the situation was decoration. And the situation is where understanding lives. So your function isn't intuition with the glow removed. It's the projection of intuition onto the one axis a closed game preserves — and the moment you move to a world with a real situation, the projection stops being enough, because the discarded dimensions come roaring back.

HASSABIS: But "the situation comes roaring back" is a prediction, and it's the prediction my whole career keeps falsifying. You said it about chess — that the machines couldn't do it because real play required something embodied. They do it. You said it about the open texture of common-sense language. The systems handle it, imperfectly, but at a level no symbolic system approached. Every time the "discarded dimension" is named, we eventually capture its effect from data, because the dimension left a trace in the world, and the trace is learnable.

DREYFUS: Its effect, Demis. Listen to your own word. You capture the effect of the dimension. You produce the behavior the situation would have produced. I have never disputed that the effects are learnable — that's the thing the deep learning revolution proved and the symbolic people couldn't, and I credit it without reservation. My claim was never that the behavior is unreachable. My claim is that reproducing the effect of understanding is not understanding, any more than a perfect recording of a violin is a violinist. And the place you find out the difference is not in the recording, which can be flawless, but at the edge — the live moment, the novel situation, where there's no recording to play back and only a being who actually grasps the music can improvise. Go has no edge, because the rules close the world. The real world is all edge.

EDO SEGAL: I want to mark something, because the reader can't see your faces and this is the first place the temperature dropped. You've found your seam, and it's clean. Demis says: name the missing ingredient or admit it does no work. Hubert says: the ingredient does no work in a closed game, which is the only place you've shown it, and the work it does is exactly the work that closed games are designed to remove. That's not two people misunderstanding each other. That's two people understanding each other perfectly and disagreeing about what a closed world is evidence for. Let me push one thread before we break, because there's an animal that's been waiting under the table since the opening, and Hubert, you keep gesturing at it. Demis brought up the trace — the idea that everything embodied leaves a learnable residue. There's a thought experiment that says the residue is never enough. Let's meet the octopus.