The Crossing
EDO SEGAL: The floor is yours, both of you. I'll speak only if the room catches fire. Marvin — you start. Ask him the thing.
MINSKY: All right, Roger, here's what I actually want to know, and it's not rhetorical. Suppose I build it. Suppose I give you a machine that does mathematics better than any human who has ever lived — proves the hard theorems, invents the new fields, and when you hand it a fresh Gödel sentence for its own system, it does the thing you say only a human can: it steps up a level, builds the bigger system, and tells you the sentence is true, and it does this reliably, forever, for every system you throw at it. What, on that day, is left of your argument? And don't tell me it still doesn't understand. Tell me what observation would be left for the word to attach to.
PENROSE: It's the right question and I'll answer it without flinching. On that day, two things could be true and I have to say which I'd believe. Either the machine is doing it by an algorithm — in which case it has a fixed formal system underneath, however vast, and there is still a Gödel sentence it can't reach, and you've just hidden the wall further out, not removed it; or the machine is genuinely not running any fixed algorithm — in which case it is no longer what I mean by a computer, it's harnessing the same non-computable physics I say you harness, and then it might well understand, and I'd grant it, gladly. So my argument isn't refuted by your machine. It's sharpened by it into a single question: is the thing computing, or not? If it computes, the wall stands however far out you push it. If it doesn't compute, you haven't built a counterexample to me — you've built a second candle, and I'd welcome it. What you can't build is a computer that crosses the line, because the line is the edge of computation itself.
MINSKY: See, that's the move I think is a trap door, and I want to push on it, because you've made the wall un-hittable by definition. You've said: if it crosses the line, it must not have been a computer, because computers can't cross the line. That's a circle. Any actual machine I build, running on actual silicon, is a computer — it's physical, it's finite, it follows rules. You've reserved the right to say, about anything I ever build, "ah, but if it really understood, it wasn't really computing." How is that different from a man who says no machine can be alive, and every time you show him one that grows and reproduces and metabolizes, he says "then it isn't really a machine"? At some point the word "machine" — or "computer" — is just a label you peel off anything that succeeds.
PENROSE: Because the line isn't drawn by me, Marvin, it's drawn by Gödel, and that's the difference between my circle and the vitalist's. The vitalist had no theorem; he just kept retreating. I have a theorem that says, precisely, for any formal system, here is a truth it cannot reach. That's not me peeling off a label. That's mathematics handing me, in advance, a specific limitation of every computer you will ever build, before you build it. So when I say "if it crossed the line it wasn't computing," I'm not protecting a prejudice — I'm reporting what the theorem guarantees. Your silicon machine is a formal system; the theorem applies; the sentence exists; the wall is there. If you ever show me a physical device with no such wall, you'll have shown me a device that isn't a formal system — and I'll be fascinated, and I'll want to know what physics it's running, and I'll bet it's mine.
MINSKY: Then let me ask the question underneath the question, because I think this is where you actually live. Roger — do you want to be a machine? Set the theorem down for one minute. When you strip it all back, isn't the engine of this whole thirty-year argument that you can't stand the idea that the thing you do when you see a proof — the most beautiful experience of your life, you've said — could be the same kind of thing a chip does? Isn't Gödel the respectable clothing on a refusal that came first?
PENROSE: [pause] That's the sharpest thing anyone's asked me tonight, and I won't dodge it. I'll give you two answers and you can decide which is real. The honest introspective answer is: no, I don't think I'd mind being a machine — I find the brain more wonderful for being physical, same as you do, I'm not reaching for a soul. What I'd mind is believing something false because it's flattering, and I've spent a career trying to believe only what the mathematics forces. The second answer, the one I'm less sure of: maybe the wonder did come first, the experience of seeing, and Gödel came after as the thing that turned the wonder into an argument. But Marvin — turn it on yourself, because it's exactly symmetric. Didn't your "it's all machinery" come first too, before any proof? Didn't you need the mind to be buildable, because you'd given your life to building it? Neither of us reasoned our way to the starting line. We reasoned our way from it. The question is only which of us is now letting the evidence move him, and I'd like to think it's both, a little, tonight.
MINSKY: [pause] That's fair. That's completely fair. Yes — I needed it to be buildable. I'd have hated to spend sixty years on a thing that was impossible in principle, and that hatred is older than any argument I have. So we're two old men who each fell in love with a starting line. The difference is I think mine leads somewhere you can keep walking, and you think yours has a wall with a view, and neither of us is going to take the other's word for what's on the far side, because neither of us can get there.
PENROSE: No. But I'll say this, and mean it: I'd rather be wrong with you than right with most of the people who agree with me. The people on my side who say "machines can't understand" mostly mean "machines are beneath us." I've never meant that. I think the machine is a glory. I just think the glory is a tool, and the candle is something else, and you think the candle is a very fine tool, and we are closer, you and I, than either of us is to the crowd that's certain. Now let me ask you mine, since you asked me yours. Marvin — you've conceded all night that the present machines confabulate, forget, hallucinate, fall off the cliff at the edge of their training. You say: missing parts, build them. So tell me the order. Which part do you build last? When you've built the memory and the self-model and the error-correction and the rich world-model and the thirty ways to think — when the society is complete and the thing does everything you do — is there a final part, the part that makes the lights come on inside, and do you know how to build that one, or are you betting it comes free with the others?
MINSKY: It comes free with the others, and I'll tell you why that's not a dodge. You're asking me to name the part that produces the inside as if "the inside" were one more component on the bench. It isn't. The inside isn't a part you add at the end; it's what a sufficiently rich self-monitoring system is like while it runs. You don't build a "feeling" module and bolt it on. You build the monitoring, the modeling, the reactivation of states, the self-model that watches itself watching — and being that, from inside, is the feeling. There's no last part with a switch on it. That's the whole content of my position: the lights aren't a thing you turn on. The lights are what running looks like to the runner. You keep asking me where I install the soul, and I keep telling you the soul is the installation, watched from within.
PENROSE: And that — "being that, from inside, is the feeling" — is the sentence I've spent thirty years unable to accept and unable to refute, so I'll just lay it next to mine and let the reader hold both. You say: be the right machine and the inside comes free. I say: you can be the right machine in every functional respect and the inside stays dark, because the inside isn't function, it's the non-computable seeing, and you can't get it free from things that don't have it. We've each named the other's load-bearing sentence. Neither of us can lift the other's. That's the Crossing, I think. Not a winner. A clean look at exactly where the floor of the argument is.
EDO SEGAL: [quietly] The room didn't catch fire. It did something rarer. Two men who disagree about the deepest thing there is just told each other the truth about why they believe what they believe, and neither flinched, and neither moved. We close after this. Final statements. The last word, each.