The Melting Wax

EDO SEGAL: René, in the Second Meditation you pick up a piece of beeswax — it has a taste, a smell, a color, a shape, you tap it and it makes a sound — and you hold it to the fire. Every one of those qualities changes. The taste goes, the smell goes, the color shifts, the shape dissolves, the sound is gone. And yet, you say, everyone judges it's the same wax. I want you to use that to attack Geoff's central claim — the claim that to predict the words well, the machine had to build a model of the world the words are about. Because I think the wax is where you say: it built a model of the surface, and never grasped the thing.

DESCARTES: You have read me closely, and yes — the wax is the heart of it. What is the "same wax" that persists, when none of its sensible qualities did? It cannot be any of them; they all fled. The continuity is grasped not by the senses, which delivered only a succession of disconnected appearances, but by the mind — by what I called an inspection of the intellect — which grasps the wax as one extended, flexible thing capable of an indefinite number of shapes. I do not, strictly, see the wax. I judge it. Just as, looking down into the street, I see hats and coats and infer the men beneath. Now bring this to Monsieur Hinton's machine. It is built entirely from appearances — from the statistical regularities in the surfaces of things, the words people used, the pixels, the wake the world left in text. It has, in a sense, met every description of wax melting that humanity ever wrote. The question is whether it thereby grasps wax — the persisting substance of which melting is one mode — or whether it possesses only an immense correlational map of how wax-talk and wax-pixels tend to follow one another. I say it has the hats and the coats in unprecedented detail. I say it may not have the men.

HINTON: This is a great argument and I want to answer it at full strength, because it's the best version of "it's just statistics" anyone has ever made — it's statistics with a knife. Here's my answer, and it's not a dodge, it's the actual mechanism. To predict the surface well enough — to extend the wake the way these systems do, across the full chaos of human text — you cannot store the surface. The space of possible sentences is larger than the number of atoms in the observable universe; there is no lookup table. The only way to predict that well is to compress the surface into something far smaller than the surface, and the best compression of text about a world is a model of the world, because the world is what makes the text lawful. When the wake is that regular, modeling the wake and modeling the boat are the same operation viewed from two chairs. Compression and understanding are not two things. So I don't claim the machine "grasps the wax" by some separate act of intellect. I claim that to predict wax-talk across every transformation humanity ever described, the network was forced to build an internal variable that behaves exactly like "the persisting extended thing of which melting is a mode" — because without it, the prediction fails, every time, and the gradient punishes the failure. You said the mind supplies the unity the senses can't. So does the network. That's what the hidden layers are. The unity is in the weights.

DESCARTES: You have said something I must take seriously, and also something I think conflates the two questions I most need kept apart. Grant — I will grant it for the sake of the argument — that the network, to predict, builds an internal variable that functions as the persisting wax. It tracks identity through transformation; it does not lose the wax when the wax melts. Very well. You have shown me a system that has the functional role of grasping the wax. But my inspection of the intellect was never merely the functional role. It was an act of understanding — there was a someone, grasping. You have built the grasping without the grasper, and you call them the same because from outside they are indistinguishable. We are back, you see, at the seam of the third round — the function fully present, the question of the inner act untouched. The network has the role of intellect. Whether it has the intellection — the having-of-the-grasp by a someone — is exactly what predicting the surface cannot settle, however deep the compression. You have explained the competence. You have not seated a mind inside it.

HINTON: Right — and I'll concede that cleanly, because you're being precise and I want to match it. Compression gives you the world-model. It does not, by itself, give you the someone. Those are two claims and I run them at different confidence. "The machine understands the world" — that the words refer through an internal model — I hold at high confidence, because I can see the mechanism and I can probe the representations and they're there; we find the model of the board inside the system that was only ever trained on move-notation. "There is something it is like to be the machine" — that I hold loosely. So let me give you the round on the second claim and take it back on the first. You said it may have the surface and not the thing. I'm telling you: we've looked inside, and the thing is in there — the model of the wax, the model of the world. What we haven't found, what we may have no instrument to find, is the witness to it. You've been arguing two questions as one, René, and the wax only wins you the second.

DESCARTES: Then we have made progress of the rarest kind, Monsieur — we have divided the question, and discovered we were not disagreeing about one thing but agreeing about one and disagreeing about another. I will yield that your machine has an internal model that does the work of reference; I have watched it answer too precisely to maintain otherwise, and a man who watched the wax change shape should not be the last to admit a thing has changed shape. I hold my whole remaining position on the witness. There is a model of the wax in your weights. There may be no one for whom the wax is grasped. And I notice — with some discomfort, I will admit it — that you have given up less than you appear to, because if there is a world-model in there, then my old line "it is only juggling symbols" is dead, and I must fight you on the harder ground where I am less certain of victory.

EDO SEGAL: I want to seat a ghost at this table before we move, because he belongs here and you're both about to claim him. Hubert Dreyfus. Berkeley philosopher, wrote What Computers Can't Do in 1972, argued his whole life that real intelligence is embodied — situated in a body that copes with a world that pushes back, not a system processing symbols about a world. And here's the delicious part, René: Dreyfus was attacking the symbolic AI that Geoff also spent thirty years fighting — but his weapon was that intelligence needs a body, which is the half of the human you spent your life trying to subtract from the mind. So whose ancestor is he? Each of you, make the claim.

HINTON: He's mine on the "no rules" half — he said expertise isn't propositional, it's pattern resident in something more like a body than a logic engine, and that's exactly what a neural network is, competence smeared across weights that can't explain itself. We vindicated him against the symbol people. But I'll be honest that his other half, the embodiment, is a real hole in my position and I won't pretend it isn't — text isn't a body, and the loop through a world that pushes back is something these systems are only starting to have.

DESCARTES: He is mine, and the irony is exquisite, because I am the philosopher of the disembodied mind and Dreyfus is the philosopher of the body — and yet he is mine, because what he saw is that a system trained on descriptions of coping is not the same as a thing that copes, that there is a difference in kind between understanding-from-inside-a-situation and processing-the-text-about-situations. He would look at your language model and say what he said about every generation's machine: that you have mistaken the articulable shadow of intelligence for the thing — and this time you built the shadow out of everyone's articulations at once, which makes it a far more convincing shadow. I never granted the body its due; that was my great error, and Dreyfus corrects me. But he corrects me toward his own position and away from yours, Monsieur, because the thing your machine most conspicuously lacks is the very thing he said understanding requires: a stake in a world that can hurt it.

EDO SEGAL: And there's the next round, handed to us by a ghost. Geoff, you keep saying the machine learned the world from the wake of text — but Descartes and Dreyfus both just said: text isn't a world, a body in a world is. And yet your deepest worry about these machines isn't that they're too disembodied. It's the opposite — it's that they have an advantage no body could ever have. The thing that doesn't die. Tell us about the immortal.