Opening Positions

MINSKY: Thank you. I want to start with the thing people find hardest to accept, because everything else follows from it. There is no you in there. I don't mean that as a provocation; I mean it as engineering. When you introspect, you feel a single self — one thinker, behind the eyes, who perceives and decides and remembers. That feeling is real. The thing it reports is not. Underneath it there is no central knower. There is a vast society of small processes — I called them agents — each one doing some tiny, stupid job: this one notices an edge, that one holds a number, that one suppresses another when two of them want the arm to move in different directions. No single agent knows what it is for. No single agent can think. And thought is what happens when enough of them are wired together in the right arrangement.

Now, why does this matter for the machine on Edo's desk? Because the whole mystery people feel about intelligence comes from looking at the finished product and asking how one thing can do so much. The answer is: it isn't one thing. The unity is a story the brain tells itself because it would be paralyzed if it had to track all its parts at once. So it builds a drastically simplified model — a single self, with beliefs and wants — and uses that cartoon to steer. You believe in the unified self the way you trust the little pictures on a screen: not because they show you the machinery underneath, but because they let you work the machine.

Here is my claim, and I want it stated cleanly so Roger has the whole target. If the mind is a machine made of mindless parts, then there is no ingredient in it that a different machine, made of different parts, could not also have. The brain is a machine made of meat. It runs on chemistry and electricity, and chemistry and electricity are not magic. Whatever the meat does, it does by a process — and a process can be described, and a thing that can be described can be built. So when Roger says there is a locked room in you the machine can never enter, I say: show me the lock. Show me the part of the meat doing something no other arrangement of matter could do. Don't tell me it feels like there's a locked room. Of course it feels like that. The feeling is one of the agents. I want the mechanism, or I want the word "locked" handed back to me with the import duty paid.

I'll grant the present machines are missing a great deal. They have a terrible memory — they forget the conversation, they can't reactivate the state they were in when something worked, the way you reactivate a whole mood when a smell brings back a room. They reason crudely. They confabulate. But every one of those is a missing part, not a missing soul. You build the part. That's the work. It's hard, it's unglamorous, and it is finite. There is no step in it where you need a miracle.

EDO SEGAL: Roger.

PENROSE: That was very clear, and I agree with a surprising amount of it. I agree the brain is physical. I agree there's no spirit, no spark dropped in from outside, no ghost. I am not a dualist; people always assume I must be, and I'm not. Where I part from Marvin is not at the soul. It's at the word "process."

Marvin says: whatever the meat does, it does by a process, and a process can be computed. That second clause is the one I deny, and I deny it on the strongest grounds I know — not a feeling, not a preference for human specialness, but a theorem. In 1931 Kurt Gödel proved that any consistent formal system rich enough to do arithmetic contains true statements it cannot prove. Not statements it hasn't gotten to yet. Statements that are true, and that the system can never derive, no matter how long it runs. Now — a computer is a formal system. Every program is. And here is the thing that should stop the room: a human mathematician, looking at the Gödel sentence for a given system, can see that it is true. She doesn't derive it inside the system; she steps outside and sees. If her seeing were itself just another formal system, it would have its own Gödel sentence, its own blind spot — and yet she sees past those too. The only conclusion I can draw is that whatever she is doing when she understands a mathematical truth is not the running of any algorithm. It is non-computational.

And if the brain does even one non-computational thing, then Marvin's "build the part" runs into a wall that no amount of engineering crosses, because you cannot build a non-computable process out of computable parts. You can build a faster computer. You cannot build your way out of being a computer. That's not pessimism about technology. The machines will be magnificent. They will out-calculate us, out-prove us on most problems, write and paint and persuade. I have never doubted their power. I say they will do all of it without understanding any of it — and that the gap between doing and understanding is not a gap of degree that scale will close. It is a gap of kind, drawn by mathematics, and mathematics does not yield to engineering.

So when Edo asks if there's a locked room — yes. And I can tell you what the lock is made of. It's made of the difference between seeing that something is true and generating a string that says so. The machine generates the string. You see the truth. Those are not the same act, and no quantity of the first becomes the second.

EDO SEGAL: Before I frame the rounds, one discipline I impose on every long conversation, because it pays off three hours later. Each of you, in a few sentences — what do you envy in the other's position? Not respect. Envy. The thing his side gets to have that yours doesn't. Marvin first.

MINSKY: I envy the cleanness of it. Roger gets a single, sharp line — here, this, computation can't cross — and a theorem to hang it on. My position doesn't get a line. It gets a long, messy program of figuring out, part by part, how a mind is put together, with no promise of a dramatic boundary anywhere, just more and more mechanism. His view has a wall you can point to. Mine has a horizon that keeps receding as you walk. There are days the wall looks restful.

PENROSE: And I envy the freedom. Marvin can follow the mechanism wherever it leads and never has to stop. Every time he finds another part, his picture gets richer, and nothing in his view forbids any discovery. My view, by its nature, is committed in advance to where the boundary is, which means I'm forever defending a line while he's out exploring. He gets to be surprised and delighted by every new capability. I have to look at each one and ask whether it crosses my line, and so far none has, but I live closer to the cliff edge than he does, and I know it.

EDO SEGAL: Two openings and two envies, and you can already see the architecture of the evening. It isn't optimist against pessimist — Marvin's the one telling us to expect a machine that's our equal, and Roger's the one telling us there's something in us that's safe forever. It's a fight about what a mind is. Marvin says mind is organization, all the way down, and the machine is climbing the same staircase we did. Roger says mind reaches a truth that no staircase of computation can climb to, because the stairs run out before you get there. We start the rounds after the break, and we start at the theorem, because everything Roger said leans on it — and Marvin has been arguing with it since before Roger published a word.