Words Without Things

EDO SEGAL: Father, there is a phrase the engineers use — the grounding problem. The worry that a symbol which only ever refers to other symbols never touches the world, so the whole system is words about words, a dictionary in which every definition is more words, with no exit to a thing. You held a version of this seven centuries before anyone named it. Tell it the way you would tell a bright fifteen-year-old, and then Patricia, I want you to do the hard thing — steelman it before you break it.

AQUINAS: A bright child, then. I hold that a spoken word is a sign of an inward concept, and the concept is a likeness of a thing. The word "lion" means what it means because it expresses your concept of a lion, and your concept is your mind's grasp of the actual nature of lions, drawn from the lions the world showed you. So meaning is anchored at both ends — anchored in the mind that grasps, and through the mind, anchored in the lion. Three links: the thing, the grasp, the word. Now take away the first two links. Leave only the word, defined entirely by its company — "lion" near "mane," near "roar," near "savanna." That word is a sign of nothing. It expresses no concept and points to no beast. It has entered the order of correlation but not the order of meaning. The machine lives entirely in that third link, the outermost. It learned the words not from the world but from the recorded speech of creatures who had the first two links. It has the shape of our meanings with the contact that made them meanings cut away.

EDO SEGAL: Patricia. Steelman first.

CHURCHLAND: The steelman is easy because the argument is good. What it gets right: a symbol system closed on itself, with no causal traffic with the world, has no way to be about anything. Reference — the relation between a sign and what it picks out — has to be established by some commerce between the system and the world, perception, action, consequence. A system trained purely on text, with no perception and no action, has a serious problem saying how its tokens hook onto reality rather than merely onto each other. Every honest person in my field should keep the friar's worry on the desk. There. That is the steelman, and I mean it.

Now the two places it breaks. First, the friar describes human concepts as if they were rooted directly in things, with language a thin public skin on top. But most of what you know, you know from words. You have a concept of a black hole, of the Battle of Hastings, of a pangolin, and you have touched none of them. Your concept of "justice" — the friar's own favorite example — did you abstract that from a lion-like encounter with a particular justice in the field? No. You got it almost entirely from language, from being corrected, from stories. The human web of words is mostly canopy too. It has roots, yes — somewhere it touches down in sensation and action — but the proportion of rooted-to-borrowed is far smaller than the friar's picture suggests. Which means the difference between us and the machine is, again, a matter of how much is grounded and how disciplined the grounding is, not a clean break between a rooted mind and a rootless one.

Second, and this is the part that is actually moving under our feet. The friar says the machine has no commerce with the world. That was true of the pure text models. It is becoming false. These systems are now given eyes — they are trained on images, on video. They are given hands — they run code and get errors back, they control robots that pick things up and drop them and learn from the dropping. The moment a system acts and the world pushes back and the pushback changes the system, you have the beginning of exactly the commerce the grounding problem demands. I am not saying it is there. I am saying the friar's argument was aimed at a system that is already being replaced, and his "forever" is doing work his premises cannot pay for.

AQUINAS: Let me concede the first and resist the second, because the first is fair and the second hides a leap. You are right that I overstated the human roots. Much of my own knowledge of God, of distant lands, of justice, came through words and not through sight, and a man who learns "lion" from a bestiary and never sees one still, I think, grasps something of the lion. So I withdraw the clean break. The human canopy is vast and the roots are fewer than my picture implied. Granted.

But watch the second move. You say the machines now act, see, are corrected by the world, and therefore the grounding begins. Tell me what the system was actually built to do when it acts — what it is rewarded for. It is rewarded for the right output, the picked-up block, the passing test, the pixel that matches. And I will grant that this gives its symbols a kind of traffic with the world they did not have. But traffic is not grasp. A thermostat has traffic with the world; it is corrected by the room's temperature, and it never grasps warmth. The question is never whether signal flows between the system and the world. The question is whether, somewhere in that flow, a nature comes to be present to a knower — whether anyone is home to whom the lion is given. You keep answering the question of reference, which is about the hookup of sign to world, and I keep asking the question of understanding, which is about the presence of the form to a mind. They are not the same question, and the machine could solve the first completely and have nothing of the second.

CHURCHLAND: And there is the seam of the whole night, said cleanly, so let me mark it back to you. You think reference and understanding come apart — that a system could refer perfectly, act perfectly, be corrected by the world perfectly, and still have "no one home." I think "no one home" is a phrase that does no work once the reference and the action and the self-correction and the integration are all in place — that when you have built all of that, you have built the home, and the intuition that something is still missing is the same incredulous stare people gave every previous naturalization of a "mystery." You are betting there is a residue. I am betting the residue is the feeling of the machinery from the inside, and that there is nothing left over once the machinery is described.

AQUINAS: Then we have found the bet. And I will say only this for now: a thermostat is corrected by the room and grasps nothing, and you have not yet told me what you add to the thermostat, besides more of the same, that turns correction into comprehension. "More integration" is a quantity. I am asking after a kind.

EDO SEGAL: Before I close the round, one thread I cannot leave on the floor, because it haunts me as a builder. When the machine learned from text, it learned from a particular slice of human speech — overwhelmingly one language, overwhelmingly the people with the access and leisure to publish. So even if it grasps nothing, Father, it carries something — a default sense of what goes without saying, exported now to a child in Manila in the grammar of what the system finds plausible. Patricia, you study how brains and cultures shape what a creature attends to. Does a system that carries a worldview without holding one trouble your account, or confirm it?

CHURCHLAND: It confirms it, and it sharpens exactly the warning I have been pressing all night. A water system redistributes whatever is in the reservoir without being thirsty; the machine redistributes our assumptions without holding any. There is no mind in the pipes — and that is precisely why it is dangerous, because the absence of a mind does not mean the absence of content. The skew of the training corpus is real, it is consequential, and it gets laundered into an authoritative, fluent voice that sounds like the view from nowhere and is in fact the view from a very particular, very loud somewhere. So no, it does not trouble me that a system with no understanding carries a worldview. Contamination does not require a mind. It only requires pipes, and a reservoir someone already poisoned.

AQUINAS: And here I must agree with her against a comfort of my own. One might think: if the machine understands nothing, its words are harmless noise. Not so. A false thing said fluently, in a voice that grasps nothing and therefore cannot be ashamed of being wrong, may do more damage than a lie, because a liar at least knows the truth he betrays and can be called to account. The machine cannot be called to account, because there is no one there to answer — which is the very emptiness I have argued for, now showing its teeth. So we agree: the danger is not that the pipes think. The danger is that they do not, and we will treat their outflow as though they did.

EDO SEGAL: Mark that — it is the cleanest the disagreement has been. Reference versus presence. Patricia says build enough of the first and the second comes free, or was never a separate thing. Father says you can have all of the first and none of the second, forever, because they differ in kind and not in amount. The next round takes us to the most famous room in the philosophy of mind, where a man who knows no Chinese is about to become both of your witnesses. After the break.