Meaning Is Use, or Meaning Is Inside

EDO SEGAL: I want to open this round with a confession instead of a question, because my best questions come out of wounds. I started in Assembler. I was raised by machine code, and for fifty years using a computer meant translation — you compressed your intention into the machine's grammar and paid a tax on every conversion. Last winter the tax went to zero. The machine met my engineers in their own language: mess, half-finished sentences, implication and all. I wrote that this was the great inversion — we stopped learning to speak machine, and the machine learned to speak human. Ludwig, the sentence I wrote was, almost word for word, your sentence: that it learned our language by learning nothing but use. Tell me whether I should be proud of that sentence or frightened of it.

WITTGENSTEIN: Both, and in that order. Be proud, briefly, because what you saw is real. The interface changed; the tax is gone; I have never doubted that a machine which tracks use can be staggeringly useful. Now be frightened, because the sentence smuggles a passenger. "It learned our language" quietly means "it has what we have when we have language." It does not. It has a model of our text. And the difference is the whole of tonight.

Watch your engineer. When she described a feature "in her own language," her words were doing what human words do — pointing. At users who do not yet exist, at a frustration she has felt in her own body, at a thing she wants. The system received the words and did the only thing it does: it produced the continuation that the company of those words, across a billion prior sentences, makes likely. It worked. And notice why it worked. It worked because human beings, for seventy years, wrote text in which intentions and code sit side by side, and because she could look at the output and judge it — right, wrong, closer, no. Every gram of the pointing, the judging, the caring whether it is right was on her side of the glass. The machine did not cross over to her. Her language, and everyone's, met her in a mirror.

FODOR: And here is where the mirror talk does its conjuring trick, so watch my hands. Ludwig says: the machine merely produced a likely continuation. Fine. Now ask the question he steps over every single time — what does it take to do that? Not to do it badly. To do it the way these systems do it: take three paragraphs of a stranger's half-formed intention and return a working program; translate an idiom and keep the joke; be told "no, more like the second one" and get it right. There is no lookup table that large. The space of conversations is bigger than the number of atoms you'd need to tabulate it. The only way to continue text that well is to compress the text into something smaller than the text — and the best compression of text about a world is a model of the world. That's not my opinion. That's information theory.

So when the wake is that lawful, modeling the wake means modeling the boat. And what is a model of the world, in a system, but a structured set of representations? Which is to say: I think Ludwig has accidentally described a Language of Thought and called it a mirror. The mirror story has to explain how a mirror corrects your reasoning, composes constraints you never stated, solves a problem posed in a form it never saw. At some point "you're talking to yourself" becomes the extraordinary claim, and "there is structure in there doing work" becomes the boring, parsimonious one.

WITTGENSTEIN: You have done the thing I warned the room about in the first minute. You heard "it produces apt continuations" and you slid, without noticing the slide, to "therefore it has a model," and from "model" to "representation," and from "representation" to "meaning." Each step felt like an inference. Each was a bewitchment. Let me grant you everything factual and deny you the conclusion. Yes — there is enormous structure in the system; human use is shot through with meaning, and the traces of meaning are everywhere in the text, so a system that captures the pattern captures a real and usable shadow of meaning. I have never denied the shadow. I deny that the shadow is the thing. A shadow cast by a form of life is not itself a form of life.

FODOR: "Shadow" is also a picture, Ludwig. You accuse me of metaphor and answer me with one.

WITTGENSTEIN: [pause] That is fair. Strike the shadow. Here is the plain version. The system has the distributional regularities of language — the thin sense of use. It does not have use in the thick sense, which is the word doing work in a life: the engineer's word answerable to a user who will or will not be helped, to a product that will or will not exist, to her own competence, which can be praised or shamed. Your compression argument shows the machine has built something efficient. It does not show that what it built is answerable to anything. And meaning, for a human being, is not efficiency. It is answerability — the standing possibility of being wrong about the world, at a cost. Show me where, in the gradient, the cost lives.

FODOR: The cost lives in the loss function. It is punished for the wrong next token.

WITTGENSTEIN: It is punished for the unlikely token, not the false one. That is the whole difference, and you of all people should feel it. A human expert who fabricates a citation stands in a relation to the truth she is violating. The machine fabricates with exactly the fluency it tells the truth, by the identical process, because it was never answerable to truth — only to plausibility. The industry had to invent a word for it. They call it "hallucination." You and I both know that word is a confession: the machine has no idea what is true, because "true" was never in its game.

FODOR: People confabulate constantly. Ask any neurologist. We don't call it hallucination when a confident man does it at dinner.

WITTGENSTEIN: A confident man at dinner can be held to account. He can be shown the document, and shamed, and he will feel it, because he is answerable. That is not a small difference. It is the entire difference, and it is public, not private — I am not pointing at a missing inner glow. I am pointing at a missing place in the practice.

FODOR: Then let me press the place where your "answerability" gets thin. You keep treating shame as load-bearing — the man feels it, he cares. But a thing can be answerable without shame. A chess engine is answerable to the rules: it cannot make an illegal move; the position holds it to account with perfect rigor and no feeling whatsoever. So answerability can be purely structural — a constraint the system is built to respect. Why isn't the model answerable, in exactly that bloodless sense, to the next token? It is held to account, by the gradient, ruthlessly.

WITTGENSTEIN: Because being held to account by the rules of chess is answerability to a game, and being held to account by the gradient is answerability to plausibility, and neither is answerability to the world being a certain way. The chess engine cannot be wrong about the world; it can only be wrong about chess, and inside chess it is never wrong, because the rules are constitutive. The man fabricating the citation is answerable to something his words are about — a fact outside the game of producing sentences — and that is the answerability I mean. Your model has the chess-engine kind, to its own game of likelihood, and not the world-kind. That is precisely why "true" and "plausible" come apart in it and not in the man.

EDO SEGAL: Let me restate the seam, because the reader can't see your faces and I want this one marked. Ludwig — you're saying the machine has the form of use without answerability, and answerability is not inside anyone, it's a feature of how we live. Jerry — you're saying that to produce use this good, the machine must have built genuine inner structure, and answerability is a red herring you can't even locate in the brain. Is that the fork?

FODOR: That's the fork, and I want to drive a wedge into it. Ludwig keeps saying answerability is "public, in the practice, not inside." But what makes a creature answerable? Something in its head. The disposition to revise, the representation of the truth conditions, the machinery of caring. You can't have a public practice of correction floating in the air with nobody implementing the correcting. His "form of life" is a level of description of a mechanism — and the mechanism is exactly what I study and he refuses to look at. He's describing the weather and forbidding meteorology.

WITTGENSTEIN: And he is describing the moves of chess by weighing the pieces. Yes, there is a mechanism — neurons, weights, something. I have never denied the mechanism. I deny that the meaning is in the mechanism, in him or in the machine, because I proved — for human beings, first — that meaning was never an inner accompaniment to our words. He wants to find in the silicon the very thing that does not exist in the skull. The tragedy is that he is half right, and the half he is right about is the half that will mislead a generation.

EDO SEGAL: Hold there — that "half right" returns when we reach consciousness, and it's the hinge of the night. The round produced something clean: Jerry says fluency this good is impossible without inner structure that means something. Ludwig says the structure is real and the meaning is a confusion you import. Next round, we name Jerry's structure precisely. The Language of Thought. John loves Mary. After this.