Opening Positions

BENDER: Thank you. I want to start somewhere unfashionable — with what language actually is, because almost every confusion in this field begins by treating language as if it were the text. The marks. The tokens. The stuff you can scrape off the internet by the trillion. It is not. Language is an activity between people. When I say something to you, I have an intention — something I want you to come to believe, or feel, or do — and I have a model of you, of what you already know, of what you'll do with my words. You, hearing me, run the machinery the other direction: you ask, what was she trying to do by saying that? Meaning lives in that joint activity. The philosophers have many words for it — communicative intent, meaning as use — pick your tradition, the convergence is the same. The text, the form, is the trace the activity leaves behind. It is the wake, not the boat.

Now. A language model is trained on the wake. Only the wake. Trillions of words of form, stripped of every situation, every intention, every speaker and listener that made those words mean anything. From that it learns — and I'll be completely fair, because the engineering is genuinely impressive — it learns an extraordinarily good model of what the wake looks like. Which words follow which, at every scale of pattern, from spelling to syntax to the rhythm of an apology or a proof. And when you prompt it, it extends the wake. Plausibly. Fluently. Next-token prediction at a scale that beggars intuition.

Here is my claim, and notice how narrow it is, because the narrowness is the strength. There is nothing in the statistics of form that reaches what the form is about. To know the word "water" is not to know which words it travels with. It is to connect the word to water — to thirst, to rivers, to the weight of a bucket. A system that has only ever met the word has the first thing. It does not have the second, and no quantity of the first sums to the second, because aboutness was never in the training signal. That's the grounding problem, and scale does not touch it. A bigger model is a better model of the wake. It is not one inch nearer the boat.

So when the box answers you in your tongue and you feel met — Edo, I'm using your word, deliberately — I don't doubt the feeling. I can explain the feeling. You are a human being, which is to say a meaning-making machine of terrifying power. For the whole history of our species, fluent language had a mind behind it, so your interpretive machinery treats fluency as proof of mind. It cannot help it. These systems industrialized the trigger for that reflex. The understanding in the conversation is real — and it is entirely yours. You are alone in the room. That is not a tragedy. But build an economy, a school, an information ecology on the opposite assumption and you will hurt people at scale — you already are — and the people profiting from the confusion are not confused. They are selling it. That's my opening.

EDO SEGAL: Alan.

TURING: That was very good, and I agree with more of it than Professor Bender may expect — and the part I reject, I reject at the root. Let me start by conceding the engineering point with both hands, because I do not want to win by pretending it away. She is right that these systems were trained on text and only text, that text is in some sense the residue of human activity, and that nothing guarantees the residue contains everything the activity contained. A careful person should hold that worry. I held a version of it myself.

Now the disagreement. Professor Bender says: there is nothing in the form that reaches what the form is about, and therefore the machine cannot mean. I want to ask the question her account steps over, which is the only question I have ever found worth asking here. How would you know? Set two correspondents behind a curtain — one a person who has touched water and felt thirst, one a machine that has only ever met the word. Ask them anything you like, for as long as you like, as cruelly and cleverly as you like. If, after all your questioning, you cannot reliably say which is which — then on what basis do you assign meaning to the one and withhold it from the other? Not on the basis of anything you have observed, for by stipulation you observed no difference. You assign it on the basis of a story you already believe about what is inside each of them. And a story about the inside, which no observation can confirm or refute, is not knowledge. It is metaphysics wearing a lab coat.

This is the whole of my position, and I'll state it as starkly as she stated hers. I am not claiming the machine has a soul, or feelings, or that it is "like us" in some warm inner way. I am claiming something narrower and harder to escape: that the thing you cannot detect cannot be the thing that matters. We grant minds to one another not by inspecting an inner light — we have never once seen another person's inner light — but on the evidence of behavior, of conversation sustained across every subject, of the capacity to follow a thread and to surprise. That is all we have ever had of each other. If we grant thought to humans on those grounds, consistency forbids us to deny it to a machine that meets the same bar. To refuse is to apply, to the machine, a standard of evidence we have never once applied to a human being — and to call that double standard rigor.

And I want to press one word of hers, because it's carrying her whole opening. Reflection. She says you met your reflection. But a mirror returns what you put in. When one of these systems takes three paragraphs of your half-formed intention and returns a working argument you had not assembled — when it composes constraints you did not state, corrects an error you did not catch — it is doing something no mirror does. A mirror that performs feats of inference is no longer well described as a mirror. At some point the description has to answer to the behavior, Professor Bender. Mine answers to it. I am not sure yours does.

EDO SEGAL: Before I frame the rounds, one discipline I impose on every long conversation, because it pays off three hours later. Each of you, in a few sentences — what do you envy in the other's position? Not respect. Envy. The thing their side gets to have that yours doesn't. Emily first.

BENDER: That's an unfair question and I'll answer it honestly. I envy the cleanliness. Alan's position is a beautiful machine — it runs on one move, the curtain, and the move is genuinely hard to beat on its own terms. My position is messier, because it has to keep dragging the world back into the room: the body, the data center, the worker who was let go, the language nobody named. There are mornings when I'd love to live inside a thought experiment that elegant and let the consequences sort themselves out. I can't. But I see why it's a more comfortable place to think from, and I'll admit the comfort is a kind of power.

TURING: And I envy the floor. Professor Bender's position has ground under it — language is between people, meaning is accountable to someone, the duties run to the humans in the room — and she can stand on that floor and push. My position has no floor. I am committed to following the argument wherever it goes, and it keeps going to places that dissolve the very specialness I might want to keep — of understanding, of creativity, perhaps of experience itself. People imagine the frightening thing about my view is the machine. The frightening thing is what it implies about us. She gets to defend humanity. I am stuck describing it, and the description keeps getting less flattering. I have some personal experience of being described, by an institution, as less than I was. I would rather do the describing carefully than have it done to me carelessly. But I will not pretend the floor I've given up wasn't a comfort.

BENDER: That may be the most honest thing either of us says tonight.

EDO SEGAL: Two openings and two envies, and you can already see the architecture. It is not that one of them loves the machine and one fears it. It's that they locate the danger in opposite places. Emily says the danger is that there's nobody there and we'll believe there is. Alan says the danger is that we'll keep asking an unanswerable question about who's there, and torture ourselves and the machine with it, instead of attending to what it does. Hold both. We start the rounds after the break, at the exact seam: the test Alan proposed to end this argument, and why Emily thinks it was the wrong test from the start.