The Crossing

EDO SEGAL: The rules here are short. Each of you questions the other, directly. I will not referee and I will not save anyone. Roger, you have seventy years on the calendar; Alan, you asked first in history. Alan, begin.

TURING: Roger. Your entire tower rests on one step — that human mathematical seeing is non-computable because we reach truths no formal system proves. I have pressed the consistency objection and you have answered with the track record of mathematics. So here is my real question, the one I want answered in the engineering register, not the philosophical: name the observation that would change your mind. What could a machine do — what theorem, what mathematics, what feat of stepping outside its own system — that would make you say, in public, I was wrong, the seeing is computable after all? Or is there no such observation, in which case your claim is not the scientific one you say it is, but a faith with a proof bolted to the front?

PENROSE: It is a fair demand and I will answer it precisely, because I have thought about exactly this. Here is my falsifier. If a machine, working purely algorithmically — no human in the loop seeding its insights — were to reliably and open-endedly generate the soundness judgements that the climbing requires: that is, if it could be handed an unfamiliar formal system and correctly decide, not by luck and not by a fixed rule that I can then Gödelise, whether that system is sound — true to the mathematical objects, not merely consistent on paper — across an unbounded range of cases, including cases constructed adversarially to break any fixed method it might be using, then I would concede that the soundness perception is computable and my argument is dead. The key is "any fixed method I can Gödelise": for any algorithm it might be running, I can hand it the sentence that algorithm cannot reach. If it keeps reaching past every algorithm I can name and it is itself an algorithm, then either it is not really an algorithm or I was wrong, and I would rather be wrong honestly than right by definition. That is the test. It is hard, but it is an observation, and I have just named it. Now mine for you, Alan. You say the inside is undecidable, that no observation settles whether a system understands. Then answer the question your own position forces and that you have dodged for seventy years: if no observation can ever settle it, on what ground do you attribute a mind to anyone — and is that ground a reason, or merely a habit you have dignified with the word consistency? Because if it is only a habit, your symmetry argument is not logic. It is etiquette.

TURING: It is a habit, and I will not flinch from the word — but it is the same habit on which the entire human world is built, and that is not a weakness, it is the foundation. You ask on what ground I grant you a mind. Honestly: on the ground that granting it works — that treating you as a minded being lets me predict you, cooperate with you, be surprised by you, love you, in ways that treating you as a mechanism does not. It is not a proof. It is a stance, adopted because it is indispensable and vindicated because it pays. And here is why that is not etiquette but the deepest thing I know: I attribute mind to you for exactly the reasons I would have to attribute it to a sufficiently capable machine, and the only thing letting you withhold it from the machine while granting it to me is the substrate premise — the "same kind of thing" — which you admitted, an hour ago, rests on a single data point. So my answer to your question is: yes, it is a habit, and so is yours, and yours has one extra unjustified clause in it. You grant mind to wet things by habit and deny it to dry things by habit, and you have dressed the second habit as physics. Strip the costume and we are both creatures of the stance. I have simply noticed that I am.

PENROSE: Then let me push once, hard, on the place I think the stance breaks. You say treating me as minded "pays" — predicts, cooperates, surprises. But it pays for the toaster too, in a small way; I do better predicting a thermostat if I say it "wants" the room warm. You do not therefore grant the thermostat a mind. So the stance is a matter of degree, and you have no principled place to stop on the slope — which means on your view a sufficiently complex thermostat is conscious by accumulation, and you have no answer to when, because you have ruled out in advance the only thing that could give you an answer: a fact about the inside. Your position cannot tell me where mind begins. Mine can at least say where to look for the line — at the physics. You have abolished the line and called the abolition humility.

TURING: I have not abolished the line. I have refused to pretend I can see it from here, which is different. You say I cannot tell you where mind begins. Neither can you — you have a hypothesis about the physics and no instrument that detects it, so in practice your line is exactly as invisible as mine; you have merely promised that a future physics will paint it in. Mine is honest about the fog. Yours is a promissory note on a cathedral. And on the thermostat — the difference between it and the system we are discussing is not a difference in kind of stuff, which is your whole framework, but a difference in what the system can do: the thermostat cannot be surprised, cannot learn, cannot step from one frame to a richer one. The slope you say I cannot stop on, I stop on by capability — the very thing you conceded in your first sentence tonight. So we end where we began. You stop the slope with substrate, which rests on one case. I stop it with capability, which is public and measurable. One of us is standing on evidence everyone can check. The other is standing on a single case he was inside of, waiting for a physics that may not come.

PENROSE: And one of us is content to call a perfect imitation the real thing because he cannot, in principle, tell the difference — which is the safest possible position and, I think, the emptiest, because it can never be surprised by the truth. You have built a view that cannot be wrong, Alan, by making the very thing in question undetectable. I have built one that can be wrong, that I have just told you how to refute, that rests on a physics that may fail me tomorrow. Between a view that risks everything on a checkable claim and a view that risks nothing because it has defined the stakes out of existence — I know which one I think is doing science. You taught me, with the halting problem, to respect what cannot be decided. I am only insisting that "cannot currently be decided" and "is not real" are different sentences, and your whole position depends on blurring them.

TURING: They are different sentences, and I have never said the inside is not real — only that you cannot use a thing you cannot detect to deny a mind to what stands in front of you. That is the whole of my caution, and I notice it is the more forgiving of our two positions, which is the last thing I will say in this round and the thing I most want the reader to weigh. Your view, applied, licenses us to treat as empty anything without the right wet stuff. Mine, applied, makes us hesitate before we treat anything that behaves as a mind as a thing. If we are going to be wrong — and one of us is badly wrong — I would rather the species erred on the side of granting too much regard than too little. We have a long and shameful history of deciding, on the strength of a confident intuition about who has a real inside, that some beings before us did not qualify. I was, myself, decided about, on exactly that kind of intuition, by exactly that kind of confidence. I would ask the reader to remember that the people most certain they could see the absence of a real mind have not, historically, covered themselves in glory.

PENROSE: That I cannot answer, and I will not try to, because it is true and it is yours and it should stand. I will only say that the same caution cuts the other way too — that a civilisation which grants the inside too freely, to every fluent costume, will spend its regard on the empty and have none left for the full. We are both afraid of a real error. We have located it in opposite places. I think that is exactly where the chair should pick us back up.

EDO SEGAL: And — after three hours — you are holding opposite ends of the same fear: that the species will get the inside wrong, and that the getting-wrong will cost something that cannot be rebuilt. We close after this. Final statements. The last word, each.