The Limits Written Into the Machine

EDO SEGAL: Alan, you are the rare person at any table who has proven, with a theorem, that there are things no machine can ever do. In 1936 you showed there is no procedure that can decide, for every program, whether it will halt or run forever. Gödel, your near-twin in this, showed any consistent formal system rich enough for arithmetic contains truths it cannot prove. For seventy years people have wielded those results as a wall around the human mind: the machine is a formal system, the machine has blind spots, therefore the mind is more than a machine. You spent your life refusing that comfort. Tell me why — and tell me whether the limits cut the way the comfort-seekers want.

TURING: They do not cut that way, and I was careful about it from the start, because the temptation is to run from "computation has limits" straight to "the mind has none," and the inference is invalid. Yes — the halting problem is real, permanent, true of any machine you can build, including any superintelligence you care to imagine; and Gödel's result is its close cousin. But notice what it does and does not show. It shows the machine cannot answer certain questions. It does not show that we can. The argument that human mathematicians "see" truths no formal system can prove assumes we are consistent and complete in a way no one has ever demonstrated — and my own life's evidence is that human reasoning is shot through with error, blind spots, undecidable corners of its own. To hold the machine's limits against it while comparing them to an idealised fiction of ourselves is to cheat. The honest accounting brings the two closer, not further apart. The limit is a fact about formal procedures, and if we are formal procedures, it is a fact about us too. I never found that thought comforting. I found it true.

SEARLE: And here, surprisingly, I am not going to use the Gödel argument against you, Alan — because I think it is a bad argument and I have always thought so, and the people who reach for it to prove the mind is non-computational are reaching for the wrong tool. My case against strong AI has never rested on what the machine cannot compute. It rests on what computation is — formal symbol manipulation — and the claim that no amount of it, halting or not, complete or not, adds up to understanding. So let me take your own deepest discipline and turn it on the question that actually matters tonight. You proved that certain questions about machines are undecidable — that there is no procedure to settle them. I want to suggest that the question of the inside — is there understanding behind the behaviour, is anyone home — is undecidable in something very like your sense. No test, no benchmark, no feat of fluency, can settle it, because the room guarantees that identical behaviour is compatible with full inner life and with none. That is what makes the hard problem of consciousness hard in a way no benchmark touches. You, of all people, should feel the shape of that. We are not facing a question we have not yet answered. We are facing a question that has no procedure of answer at all.

TURING: That is the most interesting thing you have said, John, and I am going to half-agree and watch where it leaves us, because the agreement costs us both. Yes — I think the question of the inner light may be undecidable, genuinely, in the way certain questions about computation are undecidable. There may be no observation that settles whether a system that behaves as if it understands actually does. I said as much, in my own way, in 1950: I shifted to behaviour not because I believed behaviour was all there is, but because behaviour is all we will ever have evidence of. But here is where it cuts against you as hard as against me. If the question is undecidable, then your confident "there is nobody home in the machine" is exactly as unprovable as my "there is." You have been speaking as if the absence of the inner light were a finding. On your own undecidability, it is a posture — the same posture as mine, just pointed the other way. Neither of us gets to claim the verdict. We each get to choose what to do under a permanent uncertainty.

SEARLE: There is a real difference, and it is the difference between us — but I will grant you it is smaller than I usually pretend. I am not claiming to have measured the absence of mind in the machine. I am claiming an asymmetry of evidence. With you, Alan, I infer a mind from a whole causal story — a body, a developmental history, a nervous system continuous with mine, a creature that suffers and wants. With the machine I have only the chatter, and the chatter is a compression of ours, so a system with no inner states whatsoever would emit exactly the same sentences. The testimony is contaminated at the source. That does not make the file empty — you are right that I cannot prove it empty — but it makes the inference to a mind far weaker in the machine's case than in yours, and the burden stays on whoever wants to fill the file. Undecidable in principle, yes. Asymmetric in evidence, also yes. I will hold both.

EDO SEGAL: Mark this, because it is the closest the evening has come to a joint communiqué and it took your own theorem, Alan, to produce it. You both now agree the question of the inside cannot be settled by looking at behaviour — that it is, in the deepest sense, undecidable from the outside. You part on what follows: John says the asymmetry of evidence keeps the burden on the optimist; Alan says an undecidable question makes both verdicts equally unfounded, so the honest move is to decide how to act, not what is true. That fork is the staircase in miniature — you climb by choosing under uncertainty, not by waiting for a proof that will never come. Hold it. Because while the two of you debate what is in the machine, the market has already voted on what it is worth, and it is restructuring lives on the answer. The death cross, and the promise no one is making. After this.