The Trick That Isn't There

EDO SEGAL: Marvin, you wrote a sentence I've never been able to shake. You asked what magical trick makes us intelligent, and you answered: the trick is that there is no trick. Most people hear that as deflation — a shrug, a giving-up on the mystery. I've always heard it as the opposite, as a kind of awe. Tell me which it is. And tell it to me the way you'd tell a twelve-year-old at a kitchen table who just asked you, with real worry in her face, whether she's just a robot.

MINSKY: It's awe, and I'd tell her this. Sweetheart, you asked if you're just a robot, and the word doing the damage is just. You think there are two bins — robots, which are simple and dead, and people, which are special and alive — and you're scared you got sorted into the wrong bin. But there aren't two bins. There's one amazing kind of thing, a machine made of about a hundred billion little cells, each one too dumb to know your name, and you are what they do together. The reason it doesn't feel like a machine is that machines you've seen — a toaster, a toy — have a few parts doing one thing. You have a number of parts so large it has no business working, arranged so cleverly that they argue with each other, change each other's minds, remember, and build a little story called "me" to keep track of it all. That's not just a robot. That's the most complicated and most interesting object in the known universe, and it's having this conversation. The fear was that being a machine makes you small. The truth is it's the only way anything ever got to be as big as you.

EDO SEGAL: That's beautiful, and Gottfried is shaking his head, so let me hand him the obvious objection in its strongest form. Marvin's society of mind says intelligence is the arrangement of mindless agents — and that the unity you feel, the single "me," is a model the system builds, a simplification it needs because it can't represent its own full complexity. The self, on his account, is real the way a city is real, not the way an atom is. Gottfried, that is a direct assault on the monad. Defend it.

LEIBNIZ: It is a direct assault and I welcome it, because it lets me say precisely what divides us. I do not deny one word of his engineering. The agents, the agencies, the block-finder that knows only finding and the grasper that knows only grasping — I accept all of it as a description of how the behavior is produced. He has written the constitution of the mill, and it is a brilliant constitution. But notice what he has explained and what he has not. He has explained how a system can act as if it had a single will without any part containing the will. He has explained the coherence of the output. He has not explained, and his method forbids him from explaining, the one thing I asked for: the unity of the experience. The city is real, yes — but no one experiences the city all at once, from a single point, as a single felt whole. You experience your visual field that way. Right now, the color and the motion and the shape of everything before you are bound into one scene, present to one subject, in a single instant. That binding is not the output. It is the having of the output, and his society has no place for a haver. He says the self is a model the system builds. I ask: built for whom? A model is a model only if something reads it. Name the reader, Professor, or admit the reader is the very thing your account leaves out.

MINSKY: The model isn't read by a reader. It's used by other agents — that's the whole point, and it's why the homunculus never appears. You want a haver because your grammar has a subject and a verb, so you think every experience needs an experiencer standing behind it like a noun behind an adjective. But "the binding" you keep pointing at is itself a process — a set of agents whose job is to make the outputs of other agents consistent, to flag them as belonging to one moment, to one body, so the next agents can act on them coherently. The feeling of a single bound scene is what that consistency-enforcing process is, from inside the system that runs it. There's no extra act of "having." The having is the running. You've taken a verb — perceiving — and insisted there must be a noun underneath it doing the perceiving, and then you named the noun a monad and declared victory. I've spent fifty years showing you can do the whole job with verbs.

LEIBNIZ: Then you have a universe of verbs with no one who is ever the subject of one, and I say such a universe has described everything about a mind except that there is anyone there to be it.

EDO SEGAL: I want to test which of you the machine actually supports, because we have, for the first time in this argument's history, a real specimen on the table. Marvin, the large language models are, by construction, almost exactly your society — enormous numbers of mindless units, simple weighted sums repeated billions of times, no individual operation understanding anything, the competence living entirely in the arrangement. They are your thesis in a substrate you didn't choose. Does that vindicate you, or does it expose the gap, because the thing is fluent beyond anything your symbolic agents ever managed and we still can't say whether anyone's home?

MINSKY: Both, and I'll take the sour half first because it's the honest one. It exposes that I was wrong about the parts. I thought the mindless agents had to be little symbolic processes — things that did identifiable jobs, find-block, compare, negate. The field built them out of numbers instead, weights with no individual meaning, and that turned out to work better than my kind, which stings. But the deep structure is mine, and it vindicates the thing that matters: nobody had to put the understanding in. They didn't program the restaurant knowledge or the physics of falling glasses; they wired up a vast diversity of dumb units and let the arrangement precipitate the competence out of human text. That's a mind built from mindless parts, exactly as advertised. As for whether anyone's home — I keep telling you that's a suitcase. Ask me whether it attends, whether it models itself, whether it switches ways of thinking, and I'll give you a mechanism and an answer. Ask me whether anyone's home and I'll ask you to empty the suitcase first, because I don't think the word is naming one thing.

LEIBNIZ: And there is the whole disagreement in a single refusal. He will answer every question about the mind except the one I am asking, and he calls the question I am asking a confusion. But it is not a confused question, Professor. It is the only one that does not shrink as your machines improve. Whether it attends, models, switches — all of that you will build, I grant you, and each year a little more. Whether there is someone for whom the attending is like anything — that question is exactly as open when your machine can do everything as when it could do nothing.

MINSKY: And I'll tell you why I won't chase that question even though I hear how much it weighs on you — because in fifty years of building, the questions that moved anything were always the ones I could turn into a mechanism. I asked how a mind brings a whole world of expectation to a new room in an instant, and I got frames — packets of default knowledge that fire on a cue, the restaurant arriving complete with its waiter and its bill. I asked how a mind returns to a state that worked before, and I got K-lines — structures that reactivate the very agents that were running when you last solved the problem, so remembering is re-minding, putting the society back into a configuration it held. Every one of those questions paid out in something you could build, and the field is rediscovering all of them right now, bolting memory onto its monoliths, hunting for ways to bring the right past state to bear. Your question — is anyone home — has never once paid out in a mechanism, and I think that's a clue about the question, not a clue about the answer.

LEIBNIZ: Or it is a clue about your method, Professor — which is superb at everything except noticing what it is constitutionally unable to see. You have built a magnificent net, and you conclude that whatever your net does not catch does not swim. I grant you frames; I grant you the reactivating lines of memory; I find them beautiful, and I notice they are all accounts of how the society behaves, never of whether the society is had. You say my question never pays out in a mechanism. Of course it does not. That is its entire content — that there is one thing about a mind which is not a mechanism and therefore cannot be paid out as one. You treat that as the question's disqualification. I treat it as the question's discovery. A net that catches everything except water should make you wonder about water, not declare the river dry.

EDO SEGAL: Hold that — "exactly as open when the machine can do everything" — because it returns, hard, in the round on consciousness. But first I want to make Marvin pay the bill Leibniz keeps presenting: the unity, the binding, the one-scene-at-once. There's a name for that problem, and a modern theory that says unity might be something you can measure. Next round — the one who walks inside, and whether the walking-inside can be quantified.