Opening Positions

SUTSKEVER: Thank you. I want to start with the thing that is easy to miss because it sounds so simple. These systems are trained on one task. Predict the next word. That's all. There is no module for reasoning, no part labeled understanding, no special machinery anyone designed for meaning. There is only the relentless demand: given everything so far, guess what comes next, and do it across more text than any human could read in ten thousand lifetimes.

Now, here is the claim, and I want to state it carefully because it gets caricatured constantly. To predict the next word well — not passably, well, across the entire range of human writing — you cannot get there by memorizing. The space of possible sentences is unimaginably larger than anything you could store. So the system is forced into something else. To guess the next word in a murder mystery — the sentence that finally names the killer — you have to have tracked the plot, the motives, who was where, what a clue is. To guess the next word in a physics derivation, you have to have learned some physics. The text is a shadow the world casts. To predict the shadow accurately, you are forced to build a model of the thing casting it. I've said it in one sentence and I'll stand on it: predicting the next token well means you understand the reality that produced the token.

People say, dismissively, it's just statistics. And I want to ask what they think statistics is. To compress a vast amount of data into a model that predicts it, you have to find the regularities, and to find the deepest regularities you have to understand what about the world produces them. There is no clean line where sophisticated statistics ends and understanding begins. At sufficient depth they are the same operation. The model that learned to predict human text exceptionally well was forced, in the process, to build internal representations of the things the text is about — physics, psychology, causation, the structure of an argument — because those are what the text is about.

And here is why I think this is the most important and most uncomfortable idea in the field. It is not really a claim about machines. If understanding is what you get when prediction becomes good enough, then maybe our understanding is also a very good prediction engine. Maybe the brain is also in the business of modeling reality in order to anticipate it, and the difference between us and the machine is one of degree and architecture, not of kind. John wants there to be a special substance that minds have and machines lack. I spent my career discovering, result by result, that the things we thought required that special substance — seeing, translating, reasoning step by step — were learnable by a network that had none of it. Every time, the thing we said was irreducibly human turned out to be learnable. I'm betting the rest goes the same way. That's my opening.

EDO SEGAL: John.

SEARLE: That was clear and confident and wrong in a way that's worth a whole evening, so I'm grateful for it. Let me start where he ended, with the boast that the special substance keeps disappearing. It hasn't disappeared. It was never the substance he thinks I'm defending. I'm not a vitalist. I don't think there's soul-stuff. I think — and this is the foundation of everything I'll say tonight — that minds are biological. Consciousness and understanding are caused by, and realized in, the specific machinery of brains, the way digestion is caused by and realized in a stomach. Of course a machine can think. You're looking at one. I'm a machine that thinks. I'm just a biological machine, and that turns out to matter.

Now to his actual argument, because it's elegant and it commits a single, fatal slide. He says: to predict the text well, the system must model the world the text describes. Watch the move. The text is about a world — yes. But the system has access only to the text, never to the world. It has the most exquisite map of how symbols relate to other symbols, built from more examples than any human could see, and not one of those symbols is ever laid against the thing it names. The word "water" in that machine is defined entirely by the company it keeps — "wet," "drink," "river," "blue" — and never once by water. This is the oldest distinction I know how to draw, and it is the whole game: syntax is not semantics. Form is not meaning. You can have all the form there is — every statistical regularity in every sentence ever written — and not a single particle of aboutness, because aboutness was never in the training signal. It couldn't be. It never came down the wire.

That's my room. A man inside, shuffling Chinese characters by a rulebook, producing perfect answers, understanding nothing. Ilya's machine wrote its own rulebook by tuning billions of numbers until it became superb at the shuffling. A bigger rulebook is still a rulebook. The man memorizing the whole thing and doing it in his head, outdoors, with no room at all, still doesn't understand a word of Chinese. And neither does the network, because executing a program — any program, of any size — is the wrong kind of thing to ever amount to understanding. It's shape-shuffling. Shapes are not meanings.

He says maybe we are also just predictors, and that's the move I most want the reader to watch, because it's where the whole thing turns into philosophy whether he likes it or not. When I think about my late father, my thought is about him — that specific person, fixed by my history with him, my having stood in rooms with him, the causal and perceptual threads that tie my mind to that one man and no other. When the machine produces a beautiful paragraph about "your father," nothing in it reaches out to anyone. The aboutness is supplied entirely by you, the reader, exactly the way the meaning of marks on a page is supplied by the reader and not by the paper. The machine is a device for producing strings that we make be about things. That's not a smaller version of a mind. It's a different kind of object entirely, and the fluency is precisely what hides it. So: is anyone home? My answer is no — and the better it gets at sounding like someone, the harder you'll have to work to remember it. That's my opening.

EDO SEGAL: Before I frame the rounds, one discipline I impose on every long conversation, because it pays off two hours later. Each of you, briefly — what do you envy in the other's position? Not respect. Envy. The thing his side gets to have that yours doesn't. Ilya first.

SUTSKEVER: I envy the floor. John gets to stand on something — a hard line, biology, the felt fact of his own understanding — and push. He can say "this, what I'm doing right now, this is the thing, and the machine doesn't have it," and point at himself, and it feels like bedrock. My position has no floor. I'm committed to following the mechanism, and the mechanism keeps dissolving the ground I'm standing on — the specialness of understanding, of creativity, maybe of experience itself. People think the frightening part of my view is the machines. The frightening part is what it says about us. John gets to defend the human. I'm stuck describing it, and the description gets less flattering every year.

SEARLE: And I envy the wonder. Ilya gets to stand in front of the thing and feel awe — to feel the AGI, as he says, to sense a genesis. I've spent my life as the man at the party saying "that's not what that word means," and there are mornings when that is a cold way to be a thinker. The vigilance is necessary. Someone has to keep the words honest while everyone else is falling in love. But I won't pretend the vigilance is fun. He gets the sunrise. I get the fine print. And the fine print is true, but the sunrise is warmer.

SUTSKEVER: That may be the most honest thing either of us says tonight.

EDO SEGAL: Two openings and two envies, and you can already see the architecture of the evening. It's not that one of them loves the technology and one fears it — they'd both tell you to take it seriously. It's that they locate the mystery in opposite places. Ilya says the depth of the prediction forces a real understanding into existence, and the danger is that we'll refuse to see it. John says the depth of the prediction forces only a better mirror into existence, and the danger is that we'll insist we see someone in it. Hold both. We start the rounds at the exact seam: what, precisely, did the machine learn when it learned to predict our words?