The Theorem That Draws the Line

EDO SEGAL: Roger, I want to start this round at the foundation, because your whole house stands on it. For people who have never sat with it: Gödel proved, in 1931, that any system of mathematics powerful enough to be interesting contains true statements it cannot prove from inside itself. You take that result and make a claim about minds. So tell me, as plainly as you would tell a bright fifteen-year-old, why a theorem about arithmetic tells us something about whether a computer can understand. And Marvin — I want you to do something unusual before you attack it. Steelman it. Tell us what the Gödel argument gets right.

PENROSE: Plainly, then. A computer running a program is a formal system — it has rules, and it grinds them. Gödel hands you, for any such system that's consistent, a particular sentence — call it G — which says, in effect, "this sentence cannot be proved by this system." Now think it through. If the system could prove G, then G would be false, and the system would have proved a falsehood — so it's inconsistent. If the system is consistent, it cannot prove G — which means G is true. So G is a true statement the system cannot reach. And here is the move that matters: we just saw that G is true. You and I, sitting here, followed the reasoning and concluded G is true. The system can't. We can. We did it a moment ago. Whatever we did when we saw it was not something the system can do, because if it were, the system could prove G, and it can't. So human mathematical understanding is doing something no consistent formal system does. And a computer is a consistent formal system, or it's a useless one. That's the line.

EDO SEGAL: Marvin. Steelman first.

MINSKY: I can, because part of it is just true. What Gödel got right — and it's permanent, nobody takes it back — is that no fixed formal system captures all of arithmetic truth. Any system you write down has truths it can't reach. That's real, it's beautiful, and every engineer should have it tattooed somewhere. And Roger's right about something deeper, too: a mind that could only run one fixed formal system would be horribly limited — it would hit its blind spot and sit there forever. So if you thought intelligence was one frozen rulebook, Gödel kills that idea, and good riddance. There's the steelman. Now —

EDO SEGAL: Take it.

MINSKY: Now the part where it fails, and it fails right at the word "consistent." Gödel's theorem is about consistent systems. The devastating result only lands if the system never contradicts itself. Roger needs the human mathematician to be a consistent system, so that her "seeing" outruns the machine. But human beings are not consistent. We believe contradictions all the time. We make mistakes in proofs — long proofs go wrong, whole communities accept a result for years before someone finds the hole. I am, demonstrably, an inconsistent system. And an inconsistent system can "prove" anything, including its own Gödel sentence, and the theorem says nothing about it at all. So the argument proves nothing about me, because I don't meet its one precondition. Roger's mathematician isn't transcending the formal system. She's a sloppier system that happens to get the right answer often enough to be useful — and "often enough" is not "always," which is exactly what the theorem requires.

PENROSE: I've heard this objection more than any other, and it's the serious one, so let me answer it carefully. Yes — individual mathematicians err. I err. But look at what we actually do. When I make a mistake in a proof, and someone shows me the mistake, I don't shrug; I see that it was a mistake, and I correct it, and the correction is not arbitrary — it converges. The mathematical community, over time, sorts truth from error with a reliability that no random or merely-inconsistent process could produce. A system that's just inconsistent doesn't converge on truth; it sprays in all directions. We don't spray. We home in. That homing-in is the thing I say is non-computational. The errors are noise. The convergence is signal, and you can't get that signal from "an inconsistent system getting lucky."

MINSKY: But you can, and that's exactly what learning is. Roger, you're describing error-correction and calling it magic. A community of fallible reasoners, each checking the others, throwing out what doesn't survive criticism, keeping what does — that's a mechanical process. It's selection. It's what evolution did to make you, and it didn't need to see Platonic truth; it needed a filter and enough time. The convergence you're pointing at is the filter working. You've watched a perfectly ordinary feedback loop — guess, check, discard, refine — and because the loop produces truth, you've concluded the loop must contain something that isn't a loop. That's the leap I won't make. Show me the step in the homing-in that a sufficiently clever error-correcting machine couldn't take, and I'll convert. You keep pointing at the result of the process as if it were evidence about the process.

EDO SEGAL: Let me make sure the reader feels the size of this, because the two of you just located the whole disagreement in one word and it went by fast. Roger, you're saying: we see truth, directly, and seeing isn't computing. Marvin, you're saying: there's no seeing, there's only a very good error-correcting machine, and "seeing" is what the machine's success feels like from inside. Is that the seam?

PENROSE: That's the seam exactly. And I'd add — Marvin keeps saying "show me the step the machine couldn't take." But that's precisely backwards. Gödel already showed it. For any machine you build, there's a truth it can't reach, and I can stand outside it and see that truth. You don't get to demand I find the one step; the theorem hands me a new one for every machine you propose. That's the whole force of it. It's not that I can't find the gap. It's that the gap regenerates for every formal system you name.

MINSKY: And every time you "stand outside it and see," you've just become a bigger system, with its own Gödel sentence, which you also can't see from inside. You haven't escaped formality. You've moved up one floor in a building that has infinitely many floors, and you're calling the act of climbing the stairs "seeing." I call it climbing the stairs. There's no floor where you step off the staircase into pure vision. There's just always another flight. Hofstadter wrote a whole symphony on this — the strange loop, the system that models itself and feels, from inside the loop, like something more than a loop. The feeling is the loop. There's no extra thing.

PENROSE: Then we've found, in the first round, the rock the whole evening breaks against. You say the staircase has no top and the climbing is everything. I say I am, right now, standing somewhere the staircase doesn't reach, and I got there in one step that wasn't a step on your staircase at all. Neither of us can show the other the view. That's not a failure of the debate. That's the shape of the thing.

EDO SEGAL: Let me hold you both here one more beat, because there's a piece of history under this that the reader deserves, and it's pointed. Roger, you weren't the first to make this argument. The philosopher John Lucas made a Gödel-based case against mechanism in 1961, and the field — Marvin's field — spent twenty years dismantling it, on exactly the consistency objection Marvin just made. You walked into a room where the counterarguments were already laid out and waiting. Why did you think you could carry the argument across a minefield that had already blown up the man who went first?

PENROSE: Because Lucas stated it as a slogan and I tried to state it as mathematics, and the difference is the hundreds of pages in Shadows of the Mind that nobody who waves the consistency objection at me seems to have read. Lucas said, roughly, "I can always out-Gödel the machine," and left it there, and the philosophers correctly pointed out that an individual man is not a consistent system and the argument needs more care. So I gave it the care. I didn't claim individual infallibility — I claimed the idealized mathematical reasoning we're capable of, the thing we converge toward, is not captured by any sound formal system we could know to be sound. That last clause is the heart of it, and it's where the careful version lives and the slogan died. I knew the minefield was there. I thought, and still think, the careful path through it holds.

MINSKY: And I think the careful version smuggles the slogan back in through the word "idealized." Roger, the moment you say "the idealized reasoning we converge toward," you've stopped talking about anything a real brain does and started talking about a Platonic limit — a perfect mathematician who never exists. Of course she transcends formal systems; she's defined to. But she's not in anyone's skull. The actual brains doing the converging are messy, inconsistent, finite, and — I'd bet — entirely mechanical. You've proved something about an idealization and then quietly attributed it to the meat. That's the slip. The idealized reasoner is no more real than a frictionless plane, and you can't locate her in a head.

PENROSE: The frictionless plane isn't real, and yet it's how we discovered the real laws of motion — idealizations are how physics finds the truth behind the mess, not a way of avoiding it. When I idealize the mathematician I'm doing what physics always does: stripping the noise to see the law. You'd have told Galileo he was talking about a plane that doesn't exist. The idealization points at something real in the meat — the meat is doing, imperfectly, the thing the idealization does perfectly, and the imperfection is friction, not a different kind of process. That the perfect version lives nowhere doesn't mean the meat isn't reaching for it.

MINSKY: Or the meat is doing something cruder than the idealization, and you've mistaken the target it's aiming at for the thing it actually is. Galileo's idealization was checkable — roll the ball, measure, the law holds in the limit. Yours isn't, because we can't get inside the converging community and watch it transcend; we can only watch it correct errors, which is mechanical. You've borrowed Galileo's prestige for a move Galileo never made: he idealized something we could measure our way toward. You've idealized the one thing we can't.

EDO SEGAL: Hold there — because that rock comes back in every round. Mark it: our first genuine convergence is a strange one. You both agree Gödel is real and permanent. You disagree, totally, about whether the human who sees the Gödel sentence has stepped off the machine or merely built a bigger one. The next round takes us inside the machine Marvin wants to build, and asks whether a society of mindless parts could ever do the seeing — or only ever the climbing.