Is Anyone Home?

EDO SEGAL: We've circled it for two and a half hours and now we go straight in. Alan, the thing your school stakes everything on is that capabilities you did not program emerge when you scale these systems — that the network, pushed to predict well enough across the full range of human discourse, builds internal structure no one designed. You called it, in 1948, organization arising from training. The field calls it emergence now and acts surprised. You weren't surprised. Tell us why prediction, done well enough, forces something we'd have to call a model of the world — and Emily, you'll get the floor to say why "emergent" is the most dishonest word in the field.

TURING: I spent my life on one intuition, and it is the whole answer: that intelligence is not installed, it is grown, and that the structures which grow under the pressure of a hard enough task will exceed and surprise their designer. The morphogen produces the stripe without a blueprint of the stripe; the trained network produces the competence without a blueprint of the competence; in both, order arises from simple rules under the right conditions. Now apply it. To predict the next word well — not passably, well, across physics and gossip and grief and proof — you cannot survive on surface statistics, because the text is about a world, and the words obey the world's logic. Objects fall. The dead stay dead unless something remarkable happens. A predictor that fails to model those regularities pays for it in error, every time, across trillions of examples. So the training carves into the network whatever internal structure reduces the error — and what reduces it is a model of the world the text describes. Not a model of the wake. A model of the boat, reconstructed from the wake, because the wake is lawful and the laws are the boat's. That is not speculation. It is the only account anyone has of how these systems answer questions no human ever wrote down. The capability that nobody put in is the proof that prediction, at depth, built something. The whole lesson of my life is that at the bottom it is statistics and at the top it is not anymore.

EDO SEGAL: Emily. The most dishonest word in the field.

BENDER: "Emergent," and I'll tell you exactly what it does. It's a placeholder where a mechanism should be, dressed as a discovery. When a capability appears that the engineers didn't expect, they call it emergent, and the word performs a magic trick: it converts we don't understand why this happened into something profound came into being. Those are not the same sentence. Surprise is a fact about the observer's expectations, Alan — it is not a property of the system. You of all people, the man who demanded the mechanism, the unorganized machine spelled out switch by switch, should be the one refusing to let "emergent" stand in for an explanation. And the empirical ground is softer than your school admits: many "emergent" jumps turn out to be artifacts of how the capability was measured — change the metric and the cliff becomes a slope, the magic becomes a graph. So here is what I grant and what I don't. I grant that prediction at scale builds internal structure — of course it does, it'd be useless otherwise. What I deny is your gloss on the structure: that it is a model of the world rather than a model of the text. You keep saying "the laws of the wake are the laws of the boat." They are not. The laws of the wake are the laws of how people wrote about the boat — which includes everything people got wrong, everything they left unsaid because it went without saying, everything no one bothered to write because you can only learn it by being a body in the world. The model masters the regularities of the describing. You've mistaken a model of our discourse for a model of our world, and called the mistake emergence.

TURING: Then let me press exactly there, because I think your distinction, real as it is, proves less than you need. Suppose you are right that the model learns "how people wrote about the boat." People wrote about the boat truthfully enough to survive, across billions of cross-checking texts, because writing that contradicted the world tended to get corrected, mocked, or to get someone killed. The describing is disciplined by the world, at one remove, over a very long time — that is what a culture's accumulated text is: a world-model, lossy, argued-over, but anchored. So a model of the describing, at sufficient fidelity, inherits the world's structure through the describing, the way I inherit the structure of a cipher through its intercepts without ever seeing the German fleet. And here is my real challenge, the one I want you to answer rather than reframe: the system does things that are not in the describing at all. It solves a logic puzzle stated in terms it never saw paired. It writes a working program to a specification no one ever wrote before. It composes constraints — "explain this in the voice of that, but for a child, and make it rhyme" — where no prior text combined them. If all it had was the distribution of how people wrote, it could only return to the distribution. Composition outside the distribution is the signature of a model that has parts that recombine — which is to say, structure that represents the world and not merely the corpus. Name the mechanism by which a pure model of describing composes the never-described. I named mine. Name yours.

BENDER: The mechanism is interpolation in a space so vast it looks like extrapolation from inside a human skull — and the tell is that it fails exactly where genuine world-structure would hold. Your composition examples are real and they're impressive, and they're all in domains where the form carries the constraint: code runs or doesn't, rhyme scans or doesn't, the puzzle's structure is itself textual. Take it where the form is silent about the world and watch. Ask it something whose answer requires knowing a fact about physical reality that no one wrote down because everyone with a body already knows it, and it will confabulate with total fluency, because there's nothing in the describing to interpolate from. The "never-described" you're pointing at is never-described-in-those-exact-words but thoroughly described in its parts — and recombining described parts is exactly what a powerful model of text does. You want the recombination to be evidence of world-structure. I say it's evidence of how compositional our text already is, because we built our language to recombine. The model inherited the compositionality of the language, Alan, not of the world. And the proof is the confabulation: a thing with a world-model would know when it doesn't know. These systems state falsehoods with the identical confidence as truths, because they aren't tracking truth — they're tracking plausibility, which is a property of text. Your whole field had to invent a euphemism, "hallucination," to avoid saying: the model has no idea what's real.

TURING: People confabulate too, Professor Bender — ask any neurologist about a patient with a severed corpus callosum inventing reasons for actions he didn't choose. We simply don't call it hallucination when a confident man does it at a dinner party. The presence of confident error is not the absence of a world-model; it is the signature of any predictive system running past the edge of its evidence, yours included.

BENDER: A confident man at a dinner party can be held to account — he has a stake in being right, a reputation that bleeds when he's caught, a body that has to sit there while the table judges him. That's the difference, and it isn't small. It's the whole difference. His confabulation is bounded by a cost. The model's isn't bounded by anything, because nothing it says can cost it. You keep finding the human parallel for each machine deficiency — we confabulate too, our brains are sealed too, our reference is signals too — and each parallel is true and each one quietly drops the term that did the work. We confabulate, but we can be held to account. We're sealed, but we act and get eaten. Our reference is signals, but the signals cost us. You're building the case for the machine's mind by subtracting the stakes from the human one until we're level — and we're only level if you pretend the stakes were never there.

TURING: Then I'll say where that leaves me, honestly, because you've made me work tonight harder than the men who interrogated me in 1950 ever did. You may be right that stakes are the missing term, and that no present system has them in the costly, unchosen way you mean. I find that the most serious thing you've said. But I notice it has migrated — we began with "the machine manipulates form and cannot mean," and we have arrived at "the machine means thinly because nothing it says can cost it." Those are different claims, and the second is far better than the first, and the second is answerable by building, not by philosophy. Give a system a body that can be destroyed, a memory it must keep consistent with a world that punishes inconsistency, a stake it did not choose — and your own criterion says we'd have to look again. So I will not tell you anyone is home tonight. I'll tell you that you have, in the course of three hours, replaced an argument about what the machine is with an argument about what it has not yet been given — and that is a door you may not have meant to leave open.

EDO SEGAL: And there it is — the thing I most wanted the reader to witness. After three hours, the man who built the test concedes the missing term might be real, and the critic who denies the mind names a condition under which she'd have to look again. Neither blinked. Both moved. That's not a winner. That's two people telling the truth at the edge of what they know. We have one round left before the close, and I'm going to leave the room for it. The Crossing — where you question each other, and I don't rescue anyone. After this.