Competence, Performance, and the Word "Hallucination"

EDO SEGAL: Noam, your second great gift to anyone thinking clearly here — after the poverty of the stimulus — is the distinction between competence and performance. Define it for the reader, and then I want to turn it on the machine, because I think you're about to say the machine is the strangest object that distinction has ever met.

CHOMSKY: Competence is the speaker's underlying knowledge of the language — the internalized grammatical system that determines, in advance of any utterance, what is a sentence and what it means. Performance is what the speaker actually does in real time, which is competence filtered through memory limits, distraction, fatigue, the slips and false starts of real speech. Human performance is the imperfect expression of a perfect underlying competence. When you speak haltingly, your competence still knows the rule you fumbled; that's why you can recognize your own error. Now turn to the machine, and you get something genuinely new in the world. The machine is performance with no competence behind it at all. It does not possess a grammar it consults. It has a trained disposition to continue text. So its "performance" expresses no underlying knowledge — it's surface all the way down. And that is not a flourish. It has teeth, and the teeth show in exactly the failure your industry had to name. A system with genuine competence cannot assert something that contradicts what it asserted a sentence ago, because the competence is constitutive — it constrains. A system that samples from a distribution can produce anything the distribution makes probable, including confident falsehood, including a flat contradiction of itself. You had to invent a word for this. You called it hallucination. I'd say: a thing that understood the world would not need that euphemism. The euphemism is the confession.

EDO SEGAL: Ilya, that's a hard, specific charge, and the word "hallucination" really is a tell. Answer it.

SUTSKEVER: It is a tell, and I won't pretend the word is innocent — it was coined partly to make a failure sound less like "the model has no idea what's true." Professor Chomsky is right to be suspicious of the vocabulary. But the inference from "it hallucinates" to "it has no competence" is where I get off. Two things.

First, humans confabulate constantly. Ask a neurologist, ask anyone who studies eyewitness memory or split-brain patients — the human brain invents confident, fluent, false accounts of its own behavior all the time, and we don't conclude the human has "no competence." We conclude the competence is real and the confabulation is a separate failure mode riding alongside it. The existence of a failure mode does not prove the absence of the underlying capability; if it did, your competence-performance distinction would convict humans too.

Second, and more important — the rate of these failures has been falling, fast, as the systems improve, and falling specifically when we do the things that close the loop: let the model check its work, run the code, retrieve a source, get corrected by the world. If hallucination were the signature of having no competence whatsoever, more scale and more grounding would not reduce it, because you can't reduce something from nothing. The fact that it does reduce, and reduces precisely when we connect the system to consequence, is evidence that there's a competence there that grounding sharpens. I'd put it this way: the human's competence is constitutive, as you say, so it never violates the grammar. But the machine's failures are mostly not grammatical failures — it almost never produces ungrammatical strings, which on your own theory should mean its linguistic competence is excellent. Its failures are failures of truth, of knowing-the-world, which is exactly the place where the loop through consequence does the work, and exactly the place we're now closing.

CHOMSKY: I'll grant the first point with a correction. Humans confabulate — yes — but a confabulating human can be held to account, can be shown the contradiction and feel the force of it, can revise because the competence is there to be appealed to. That accountability is not a small difference; it's the difference. And on your second point — that grounding reduces the failure — I think you've actually conceded my case while appearing to rebut it. You're telling me the system gets less unreliable when you add the loop through the world that it didn't have. Of course it does. That's my whole point: the understanding was never in the text-statistics; it has to be imported from contact with consequence. Every gain you describe is a gain from giving the system more of what I said it lacked. You're not refuting me. You're funding my account with your own results.

SUTSKEVER: That's a fair shot, and I feel it — let me sit with it a second and then give the honest answer rather than the clever one. I think you're right that grounding through consequence is essential and that pure text was never going to be enough — I've said as much; it's part of why I think we're past the age of pure scaling. Where I still disagree is the word "import." You speak as if understanding is a substance that lives outside the system and gets piped in from the world. I think understanding is a property of the model that the world's feedback refines, the way a sculptor's chisel refines a form that's emerging from the stone. The feedback doesn't carry the understanding in. It carves error away from an understanding the prediction objective already started to build. We may be describing the same elephant. You're holding the trunk and calling it import. I'm holding the leg and calling it refinement.

EDO SEGAL: And there's our richest convergence yet, so mark it carefully, because it's the one that could survive the night. You both now agree that contact with consequence — the loop through a world that corrects you — is essential, and that pure text-statistics was never the whole story. You disagree about whether that loop imports understanding from outside or refines an understanding the model began to build from the inside. Noam: import. Ilya: refinement. That is a beautiful, narrow, possibly empirical seam, and it's the third real convergence we've found. Hold it. We're an hour and a half in, and I want to climb. I want to lift this off the workbench and onto the staircase, because there's a reader out there — a parent, a kid — for whom none of this is academic. After the break, the river, the death cross, and what it means to be the person standing on the bank.