The Theorem You Both Claim
EDO SEGAL: I want to start this round with a confession instead of a question, because the best questions I know come out of something that hurt. I am not a mathematician. I am a builder. And there is a particular humiliation in standing at the foot of a theorem you cannot follow line by line and knowing that everything important to your life is being decided up there, in notation you never learned. Gödel's incompleteness theorem is that, for me. I can follow the shape — there are truths a system can contain but cannot prove about itself — and I cannot verify a single step. So I am asking the two men who can. You both claim this theorem. Alan, it is the floor of your wager; Roger, it is the blade of your refutation. The same proof. So tell me, slowly, and tell the reader who is exactly where I am — what did Gödel actually show, and how can it possibly mean both of the things you each need it to mean?
TURING: I'll give the floor first, since I'm the one resting weight on it. Gödel took the grandest dream in mathematics — Hilbert's dream, that every mathematical truth could be ground out mechanically from the axioms, that truth and provability were the same thing — and he killed it. He built a sentence inside the system that refers to itself and says I am not provable here. If the system could prove it, the system would prove a falsehood and be inconsistent. So if the system is consistent, the sentence is true and unprovable. There are true things the machine cannot reach. That much Roger and I agree on completely, down to the comma. My own halting result is the same fact in a different coat: there are questions about machines that no machine can decide. We discovered the walls together, from opposite sides of the same room.
Here is where we part. Roger says: the human sees the sentence is true, so the human is outside the walls. I say: look harder at the word "consistent," because the whole trick is hidden there. Gödel's theorem says a consistent system cannot prove its own Gödel sentence. It says nothing about an inconsistent one, or about a system whose own consistency it cannot establish. And we have no idea whether the formal system that is a human mathematician is consistent. We have excellent evidence that it is not — we contradict ourselves, we are fooled, we accept bad proofs for decades. So when Roger "sees" the sentence is true, what he has actually done is assume his own consistency and read the truth off that assumption. But a machine can do that too. A machine can be built to assume its own consistency and assert its Gödel sentence — and it will sometimes be wrong, exactly as we are sometimes wrong. The seeing is not a window onto a Platonic heaven. It is a bet on yourself, and machines can place that bet, and lose it, the same as we do.
EDO SEGAL: Let me restate that, Roger, to make sure the reader feels the cut before you answer it. Alan is saying: your famous "seeing" is just you trusting that you're consistent — and that trust is sometimes misplaced, and a machine can be given the identical trust, and then the gap between you vanishes. The thing you call insight, he calls an unverified assumption with good PR. Take it apart.
PENROSE: He has stated the standard objection precisely, and it is the strongest one there is — Marvin Minsky pressed it at me for years, and I have spent hundreds of pages on it, so let me give the real answer rather than the slogan.
Yes. Individual mathematicians err. I have erred. But watch what you are actually claiming when you say "humans are just inconsistent systems." An inconsistent formal system is not a slightly unreliable truth-machine. It is useless — from a contradiction you can derive anything, every statement and its negation, the whole edifice goes to mush. If the human mathematical community were genuinely an inconsistent formal system, mathematics would not be a tower of reliable, accumulated, cross-checked truth that has stood for millennia and lets us build bridges that do not fall down. It would be noise. The errors Alan points to are corrected — and corrected not by a bigger formal system, which would have its own unreachable sentence, but by something that keeps stepping outside whatever system it is currently in and seeing further. That stepping-outside is the phenomenon. You cannot explain it by a fixed algorithm, because for any fixed algorithm I can hand you the sentence it cannot prove and that the mathematician, given time, comes to see. You can say "the human is also unsure of her consistency." Fine. But she keeps being right in a way no single consistent algorithm can keep up with, and that reliability, across the whole self-correcting enterprise, is the evidence. Not a proof. Evidence — of the strongest kind I know.
TURING: But Roger, "keeps being right" is doing all the work and you have not earned it. The community converges, yes — and a learning system converges too. That is precisely what these machines do: they are wrong, they are corrected by the world and by each other, they update, they converge. You have described the mechanism of machine learning and called it a refutation of machine learning. The self-correcting community is not evidence of non-computation. It is the best argument I know for computation — a population of fallible learners, errors filtered by selection, converging on what works. I described that in 1948 and 1950 as the way to build a mind. You are pointing at it and saying it proves a mind cannot be built that way.
PENROSE: Because convergence on what works is not convergence on what is true, and that is the entire distinction your account cannot see. A learning system converges on what reduces its error against the data. Mathematical truth is not in the data — the Gödel sentence is, by construction, not derivable from anything in the system. The mathematician reaches it anyway. Show me the training signal for a truth that is provably absent from every formal consequence of the axioms, and I will retire the argument tonight.
EDO SEGAL: I want to mark something, because the reader can't see your faces and that was the first exchange where neither of you gave an inch. You have just located the seam of the whole evening with surgical precision, and it is one word wide. Alan, you say the human is a learner converging on what works, and the machine is the same kind of learner. Roger, you say the human reaches what is true, and truth is exactly the thing not in the training signal. Works versus true. Hold that — it comes back in every round. Before we leave the theorem, Roger, the harder version of Alan's point, because I won't let you off it. He's not only saying you might be inconsistent. He's saying that even your sense of seeing — the felt certainty, the "it arrives before the proof" — is exactly what a sufficiently complex computation would feel like from the inside, with no way to tell the difference. How do you answer the charge that your evidence is a feeling, and feelings are not measurements?
PENROSE: By agreeing that a feeling is not a measurement and then asking him to live by his own rule. Alan wants behaviour as the only evidence. Very well — the behaviour of the mathematical community over three thousand years is the data, and that behaviour is the production of reliable truth about objects that are not in any corpus, by a method that for any fixed algorithm outruns it. I am not asking you to trust my shiver, Alan. I am asking you to explain the public, checkable, historical record of mathematics by a computational process, and I claim it cannot be done, because Gödel guarantees that for any such process there is a truth it misses and we do not. The feeling is private. The track record is not.
TURING: Then we have found the real disagreement, and I am almost grateful. It is not about consciousness yet. It is about whether mathematics is discovered or constructed — whether your Gödel sentence names a truth waiting in a Platonic heaven, or whether "true" just means "what consistent extension of our practice we will eventually be driven to." If the second, my machines are doing what you do. We have to go there next, because your whole blade depends on the heaven being real.
EDO SEGAL: Which is exactly where we go. Mark the convergence first — and it is the strangest convergence I've refereed: you agree, completely, on what Gödel proved. You disagree only on what it is proof of. That fork is the next round. Is the mathematician seeing something real, or building something useful? After this.