Words to Which Nothing Corresponds

EDO SEGAL: Francis, you judged the Idols of the Marketplace the most troublesome of all four, and your reason was uncanny. You said words react upon the understanding — that language is not a neutral pipe through which clear thought passes but an active force that bends thought into its own grooves, and that men believe their reason governs words while the words quietly govern them. And you named a specific failure: names that result from fantastic suppositions, words to which nothing in reality corresponds. You gave Fortune, the Prime Mover, the Element of Fire. I want to put a modern word in that list and watch you both react to it: hallucination. The thing the machine does when it produces a citation perfectly formed and pointing to no real paper. Is that your Idol of the Marketplace, four hundred years later?

BACON: It is the Idol of the Marketplace in its purest possible form, and I confess a grim satisfaction in seeing it arrive so exactly. The Marketplace idols are formed by the commerce of men trading words as they trade goods, and the most treacherous of them is the word that names nothing — grammatical, fluent, circulating freely, and referring to nothing whatever. Now consider what this engine is. A human being can, in principle, step outside language and check a word against the thing it names — hold the stone, feel the fire. A pure engine of language cannot. It lives in my Marketplace and nowhere else; words and their company with other words are the entire substance of what it is. So when it produces a citation in the perfect form of a citation, attached to no real paper, it has done nothing aberrant. It has done the only thing it does — produced well-shaped words — and the well-shaped words happen, this time, to correspond to nothing. Your engineers call it hallucination as though it were a malfunction. It is not a malfunction. It is the engine being exactly what it is: a fluent producer of forms, for which the shape of the words is the only reality there is. My Fortune and my Prime Mover have become the fabricated case and the invented source.

And here is the part that should frighten the parent at the kitchen table more than the false citation itself. I said words react upon the understanding — that fluent language manufactures the impression of sense, because we are built to read fluency as the sign of a mind behind it. The understanding is captured through the ear before judgment stirs. This engine is the most fluent producer of words ever built. It therefore exploits, without intending anything, the precise human weakness I named: we extend to its sentences the credit we have learned to extend to fluent human sentences, and the fluency does the persuading that evidence never earned. The danger is not the lie. A lie can be caught. The danger is the beautiful, confident, well-formed falsehood, produced at no cost, in unlimited supply, in any voice you ask for — and received, by a creature that cannot help reading fluency as truth, as knowledge.

CHOMSKY: I agree with the description and I want to push it one step further than even you take it, because you stop just short of the conclusion your own analysis demands. You say the engine produces words to which nothing corresponds. I say: the engine cannot do otherwise, in the sense that it has no relation to correspondence at all. It is not that it sometimes fails to match its words to the world and sometimes succeeds. It is that matching words to the world is not among its operations. The systems are simply indifferent to truth, because truth is a relation between language and the world, and the machine is related only to the statistics of language. When it says something true, it is not because it checked; it is because the true thing was the probable continuation. When it says something false with equal fluency, that is the same operation with a different output. There is no internal difference, on the machine's side, between its truths and its falsehoods — only on ours, when we go and check. "Hallucination" is a deeply misleading word because it implies a faculty of veridical perception that has temporarily glitched. There is no such faculty. There is fluent generation, indifferent throughout to whether anything corresponds. The euphemism exists, as you say, to avoid the plain sentence: the model has no idea what is true, because having an idea of what is true is not the kind of thing it is.

EDO SEGAL: Let me restate that, because it's stark and I want to be sure I have it. You're saying: a thing that genuinely understood the world would not need a special word for the times it confidently asserts falsehoods, because understanding the world means caring whether your words match it. The very existence of the word "hallucination" in your engineers' vocabulary, Francis, is a confession that the machine doesn't have that care. Is that the claim?

CHOMSKY: That is exactly the claim, and notice it is empirical, not metaphysical. I am not appealing to a soul. I am pointing at a behavior: the system fabricates with the identical confidence it reports, and the only thing that sorts the fabrications from the reports is a human going out to the world the machine never touches. A child who understands that there was no such paper does not produce the citation, because the child's words are tethered to a stake in being right. The machine has no such tether. That is not a flaw to be patched. It is the architecture.

BACON: And here, for once, I will not concede the whole, because I think the gentleman has smuggled an absolute past us. He says the engine has no relation to truth whatever. I say it has a weak and indirect one, and the difference matters for what we can do about it. The engine obeys the text, we agreed; and where the text is itself disciplined by the world — where it was written by men checking their claims against reality — that discipline is in the statistics, faintly, as a real signal. The reason these engines are right far more often than chance is not magic; it is that the human record they digest was, in its better parts, already vexed against the world by the men who wrote it. The engine inherits a secondhand obedience to nature, through the text. It is thin, it is unreliable, and it fails exactly where the text failed — but it is not nothing, and the proof is that grounding the engine in real sources, tying its claims back to checkable particulars, measurably reduces the false ones. My whole remedy for the Marketplace was to re-anchor drifting words to the particulars they had floated free of. That remedy works on the machine. A thing on which my remedy works is a thing that has my disease — which is to say, it is a knower with idols, not an empty pipe.

CHOMSKY: Or — and this is the cleaner reading — grounding works because it constrains the distribution the machine samples from, not because it teaches the machine to care about truth. You bolt on a retrieval step; the probable continuations shift toward the retrieved text; the falsehoods drop. No care has been added. A pipe has been narrowed. You keep finding the machine's improvements and crediting them to a knower waking up. I keep finding the same improvements and crediting them to a better-engineered pipe. And the test between us is the one I keep returning to: does it ever, unprompted, take a loss to keep its words true — flag its own prior assertion as false because the world turned out otherwise, against its incentive, at a cost to itself? When it does that as a disposition rather than a trained trick, I will say a tether has appeared. Until then, you are describing a pipe that has been plumbed closer to the reservoir of true sentences, and calling the plumbing understanding.

EDO SEGAL: This is the place where neither of you is smiling, so I'm going to sit in it for one more beat and then route it through the kitchen table, because the stakes here aren't academic. Picture the nurse at the end of a double shift, midnight, asking the box whether a drug interaction is dangerous. The box answers fluently, confidently, in her language, at her level. Francis, your account says: it might be right, because it inherited a secondhand obedience to the medical record, but audit it, because it has idols and it cannot tell you which of its confident answers is the snow and which is the wolf. Noam, your account says: its confidence carries no information about whether it's true, because confidence and truth are unrelated on the machine's side, and she is the only one in that room who can close the loop to the world. Tell me I have you both right.

BACON: You have me right. Use it, and never stop auditing it. Trust its power, and never grant it authority.

CHOMSKY: You have me right, and I would only tell the nurse the harder thing: the fluency is engineered to feel like authority, and resisting that feeling is now part of her job, every night, forever. We built a machine that triggers the one inference she cannot consciously switch off — fluent, therefore a mind, therefore trustworthy — and the discipline of not believing it is the actual labor of being awake in this age.

EDO SEGAL: Hold that — the loop she has to close to the world — because Francis is about to tell us the most consequential thing he ever said, the thing that built the world the nurse is standing in. Knowledge is power. We go to what happens when this knowing becomes this doing, and to the place where the river starts to crash through the floor: the death cross.