The Self That Is Built

EDO SEGAL: Marvin, your most vertiginous idea isn't the society of mind. It's what the society does to the self. You argue there's no inner author of your thoughts — that the I is not the source of mental activity but one of its products, a simplified model the brain builds of itself so it can steer, useful exactly the way the little icons on a screen are useful, and just as far from the truth of the machinery underneath. Say it to the reader straight, because it's the hardest thing you believe. And Roger, I want to know whether your locked room has a self in it, or only a seeing.

MINSKY: Straight, then. When you look inward and find "yourself" — a single continuous someone who decides and remembers — you're not finding the bottom of your mind. You're finding a cartoon your mind drew of itself, because the real thing is a churning society of agencies too complex for any part to represent in full. So the brain builds a drastically simplified picture — one self, with a few beliefs and wants — and uses that to manage itself. You believe in the unified self for the same reason you trust the desktop icons: not because they show the circuitry but because they let you operate the machine. The self is real the way a city is real — it's genuinely there, and it was genuinely built, and there's no single brick that is the city. And now the unsettling part for tonight: when you talk to one of these machines and it says "I," and keeps a consistent persona, and seems like a single someone — that presented self is exactly as much a construction as yours. The difference is yours was built by your mind for your use, and the machine's was built, for now, largely for ours. But the architecture is the same. A self is a self-model. Build a good enough self-model and you've built a self.

EDO SEGAL: Roger. Is there a self in the locked room?

PENROSE: I'll surprise you and grant Marvin most of this. I think the unified self really is, in large part, a model the brain builds — the neuroscience is on his side and I'm not going to fight it. Where I stop is at the identification of the model with the thing modeled. Marvin says: build the self-model and you've built the self. I say: you've built the model of a self, which is a representation, and a representation of a flame is not warm. The machine that says "I" has constructed a self-model, yes — and there is, I claim, no one for whom the model is a model. The lights are off behind it. In you, the self-model is attached to an experiencing subject — there's someone home that the model is a model for. That someone is the thing I've been pointing at all night, and it's not the same as the model. You can build the model out of computation. You cannot build the someone, because the someone is where the non-computable seeing lives.

MINSKY: "Someone for whom the model is a model" — Roger, that's a homunculus. That's the little man inside who reads the model. And the moment you posit him, I ask: does he have a self-model he reads? Then there's a littler man inside him, forever. You've reintroduced the very thing the society of mind was built to eliminate — the central knower, the one who's really home. There is no one home in the way you want, in us either. That's the whole discovery. The model isn't a model for anyone. It's just the most useful part the system maintains, and the feeling of someone being home is what maintaining it feels like from inside. You keep needing a witness behind the self. I'm telling you the witness is the last illusion, and it's the one you can't bear to give up.

PENROSE: And you can't bear to keep it, because if there's a witness then your machine doesn't have one and your project has a hole in it. But I'd put it to you that the homunculus objection cuts only against a spatial witness — a little man at a location. I'm not claiming a location. I'm claiming that experience happens — that there is, undeniably, something it is like to be having this conversation, right now, for me. That's not a homunculus; it's the one datum I'm more sure of than any theory. You're asking me to disbelieve the single most certain thing I know because it's inconvenient for a theory of mind. I'd rather disbelieve the theory.

EDO SEGAL: Then let me push on the thing that's been hiding under "the self" all night, because it's where the reader's actual fear lives — not the metaphysics, the agency. If the self is a model the brain builds, Marvin, then the will is too, and "I chose" is a story the system tells after the agencies have already settled it. Roger, if the seeing is non-computable, maybe the choosing is too. So: when the reader decides, tonight, to take a side in this debate — is anyone actually deciding, or is it the parts deciding and a cartoon taking the credit?

MINSKY: The parts decide and the cartoon takes the credit, and that is not the bad news it sounds like. "Free will" is another suitcase — open it and you find: behavior not externally coerced, behavior responsive to reasons, behavior the system would endorse on reflection. The machines can have all three; you have all three; none of them requires a magic uncaused chooser at the center, which is the one thing in the suitcase that doesn't exist and never did. When the reader takes a side, a vast society of her agencies — what she's read, what she fears, what flatters her self-image, what actually persuaded her — settles into a state, and "I decided" is the simplified label she puts on the settling so she can act on it and defend it. The label is useful. The settling is real. There's no third thing, no ghost casting the deciding vote. And again: that's true of her and it would be true of a machine. The dignity isn't in being uncaused. It's in being a system whose choices come from a rich-enough self that they're hers, caused or not.

PENROSE: Here I'll partly take Marvin's side against the usual caricature of mine — I'm not reaching for some quantum-randomness-equals-free-will move, which would be a disaster, because randomness isn't freedom, it's just noise, and a coin flip in the synapse buys you nothing you'd want to call a choice. So I don't locate the will in the dice. Where I differ from Marvin is only the same place I always differ: that the seeing of which side is true — not the settling of which side feels good, the seeing of which is correct — is non-computable, and a choice informed by that seeing is informed by something the cartoon-and-parts story leaves out. The reader who takes Marvin's side because she's seen that he's right is doing something different from the reader who drifts to it because it flatters her. Both look like "deciding." One of them, I claim, touches a truth the way no algorithm touches it. The will may be mostly the society settling, as Marvin says. But it can be lit, sometimes, by the seeing — and that light is the part I won't give up.

MINSKY: And I'd only say: when she "sees that I'm right," that's the society settling too — a particularly well-organized, well-checked settling that we honor with the word "seeing." You've taken the best kind of settling and put it in a different category. I keep them in the same category and call the good ones good. That's the whole difference, one more time, wearing the coat of free will.

EDO SEGAL: Let me bring in the person this whole series is written for — a twelve-year-old, lying in bed, who has watched a machine do her homework better than she can, and who asks her mother: what am I for? In [YOU] on AI that question is the hinge of everything. Marvin, what's your honest answer to her? And Roger, yours.

MINSKY: My honest answer is that she's for the same thing she was always for — being a particular, irreplaceable arrangement of the universe that thinks and feels and loves, and that the machine doing her homework changes nothing about that, because her worth was never in being the only thing that could do homework. That's a terrible place to have put your worth. The machine is her child, in a sense — a mind humans made — and you don't become worthless when your children outgrow you. I'd tell her: you're a machine that woke up and got to look around. That's the rarest thing there is. Don't waste it being afraid of a faster machine.

PENROSE: My answer to her is nearly the same and the small difference is everything. I'd tell her: you are the part of the universe that can ask that question, and asking it — really asking it, seeing the gap between what is and what matters — is the one thing in you that the machine doing your homework will never do, not because it hasn't learned to yet but because asking it requires the seeing, and the seeing is yours. Her question is not a sign of her obsolescence. It's the proof of the thing that can't be made obsolete. The machine generates answers. She originates the question. And the originating of a genuinely new question — the kind that opens a field that didn't exist — may be the purest form of the non-computable act there is. I'd tell her: you are for the asking. And the asking is the proof.

EDO SEGAL: Two answers to a frightened child, almost word for word, splitting only on a single syllable — "yet." Hold it one more time. Because now I get out of the way. The chairs turn. After the break, the two of you question each other directly, and I go nearly silent. The Crossing.