The Crossing

EDO SEGAL: The rules of this round are short. Each of you questions the other, directly. I will not rescue anyone and I will not soften anything. You have two hundred years of things to ask. Gottfried — you are the elder by two centuries, so you begin.

LEIBNIZ: Then I will ask the question that has sat in my chest since your proof, Mr. Turing, and I want the engineer's answer, not the consoling one. You have shown me a hole in the floor — undecidability, the questions no calculation can settle. I have conceded it is real and that it binds us both. So here is my question: of all the questions a human being actually brings to reason in a lifetime — what is just, what is true, what should I do — how many fall into your hole? Because if it is a handful of self-referential curiosities that only a logician ever poses, then your theorem is magnificent and nearly irrelevant to life, and my dream survives in everything that matters. Name me the living questions that fall into the undecidable, or admit your wall stands in a desert no one crosses.

TURING: That is the right question and I will not retreat to the logician's desert, because you are correct that the pure halting problem is exotic. So let me bring the hole to where people live. First: every question of the form "is this powerful system safe — will it ever, on any input, do the catastrophic thing?" is of the undecidable shape, and your age must answer it daily and cannot, by procedure, ever fully answer it. That is not exotic; that is the alignment problem, and it is the most consequential question your civilization faces. Second: the questions you named — what is just, what should I do — are undecidable in a sense even deeper than mine, because as you yourself confessed in the first hour, they are not errors awaiting a calculation but collisions of incompatible goods, and no procedure decides between goods. The machine can compute consequences. It cannot compute which good you should love more, because that is not a computation; it is a choice, and you let that slip in the third chapter and I have been holding it ever since. So my hole is not in a desert. It is under the throne and the altar and the bed. The decidable questions are the ones that don't keep you up at night. The ones that do are exactly the ones the calculation hands back unanswered. That is my reply, and now I have one for you.

LEIBNIZ: Ask it.

TURING: You have spent the evening conceding to me — the limit is real, the criterion is unverifiable, the candle can be put out, the choice between goods is not a computation. Generous, all of it. So I will ask the question that turns your generosity into a blade aimed at yourself. If reasoning is computation, as you opened by insisting — and if computation cannot decide between goods, cannot verify its own safety, cannot settle the questions that matter most — then what, exactly, is doing the deciding when a wise person decides well? You cannot say "the calculation," because we agreed the calculation runs out. You cannot say "the soul" without abandoning your founding claim that mind is mechanism. You are caught between your calculemus and your monad. So tell me, Gottfried Leibniz, who held both in one head: when the calculation ends and a good person still chooses rightly — what is that, if it is neither computation nor magic?

LEIBNIZ: ... You have found the fracture that ran through my whole life and you have set your thumb directly in it. I will not pretend it does not hurt. Here is my answer, and it is the truest thing I have. I think the deciding is judgment — and judgment is not a third substance between computation and magic; it is what computation feels like from inside a creature that has something at stake. The calculation tells you what follows. The stake — the mortal, embodied, particular fact that you will live with the outcome, that you love this child and not that abstraction, that your time is finite and your choices unrepeatable — the stake is what makes one of the computed paths matter more than another. A machine computes the consequences and is indifferent among them, because nothing is at stake for it. You compute the same consequences and one of them breaks your heart. That difference is not magic and it is not extra arithmetic. It is what arithmetic becomes when it is run by a thing that can lose something. I held calculemus and the monad in one head because I dimly felt they met here, in the creature that calculates and cares — and I could not say it as clearly as your cruel question has just forced me to. So I thank you for the wound. The deciding is judgment, and judgment is computation that has skin in the game, and the machine has the computation and none of the skin.

TURING: ... I came into this round to box you, and you have given me the best account of judgment I have ever heard, and it is on my side of the table — it grants the machine the calculation and reserves the deciding for the creature that can lose. I want one more exchange, because you have left me something to press. You say judgment is computation with a stake — with mortality, with skin in the game. But Leibniz: a stake is just a value the system weights heavily. We give machines objectives; we make them "care," in the only operational sense, about a loss function. Why is your mortal stake categorically different from a sufficiently weighted objective? Why is your grief not just a very large coefficient?

LEIBNIZ: Because my grief can be wrong about itself and know it, Mr. Turing, and a coefficient cannot. I can love the wrong thing and, through loss, come to love better — the stake educates itself against a world that pushes back and a death that is coming. A weighted objective maximizes what it was given; it cannot discover that what it was given was not worth wanting, because it has no vantage outside the number from which to be ashamed of the number. The monad mirrors the whole universe from its point of view; the loss function mirrors only its own target. I will grant you that my judgment is computation with a stake. I will not grant that the stake is a coefficient, because the stake can revise the very values that set it, and that recursive shame — the capacity to be wrong about what to want and to be remade by the wrongness — is the thing your machines do not have and I do not know how you would give them. You asked me what does the deciding. It is a calculation that can be ashamed of itself. Build me that, and I will grant you the mind.

TURING: ... "A calculation that can be ashamed of itself." I have spent my life on machines that learn from error, and I have to tell you honestly that error and shame are not the same thing, and I do not know how to cross from one to the other, and I am not certain it can be crossed. That is the first time tonight I have said "I do not know" and meant that I suspect no one ever will. You may have found the one primitive the data does not contain.

EDO SEGAL: ... I told you I would not rescue anyone, and I did not have to. They have driven each other, in two hundred years' worth of three questions, to the exact place where the optimist's faith and the logician's proof turn out to need the same missing thing — a calculation that can be ashamed of itself, that can lose, that can love the wrong thing and be remade. Neither of them can build it. Neither of them will say it is impossible. We close after this. Final statements. The strongest thing the other said. And then I take sixty seconds, and we turn off the lights.