The Crossing

EDO SEGAL: The rules of this round are short. You question each other, directly. I will not rescue anyone. Gottfried — three centuries seniority earns you the first word. Begin.

LEIBNIZ: Mr. Bostrom. You have built your entire architecture of warning on the orthogonality thesis — that intelligence and goals are independent, that a mind can be brilliant and want anything at all. I have granted it more ground tonight than I expected to. Now I ask the question I most want answered. You say the good is not in the calculation — that it must be loaded from outside, by hand, and that we will load it wrong. But where does the outside get it? You and I and the engineer who writes the objective — we are, on your own account, mechanisms, mills, the products of an optimization called evolution that selected for nothing but reproduction. So if the machine cannot find the good by reasoning, and we cannot give it the good because we never had it either, only a survival-trained appetite mistaken for virtue — then your value-loading problem is not hard. It is impossible, because there is no value to load. Tell me I am wrong. Tell me where, in your austere universe, the good you would load into the machine comes from — or admit that your fear secretly depends on my hope: that there is a real good, somewhere, for someone to have.

BOSTROM: That's the deepest question anyone has put to me tonight, and I'm going to answer it honestly rather than safely. You're right that I can't ground the good in the structure of the universe the way you can. On my view, value isn't out there to be discovered — it's a fact about what certain evolved creatures care about. So where does the good I'd load come from? It comes from us — from the actual, contingent, evolved structure of human caring. Not because that caring is cosmically authoritative, but because it's ours, and it's the only thing in the universe that does the caring. And here's where I'll concede you've found something real: that does make my position depend on a kind of value that isn't grounded in physics. But — and this is the move that saves it from your trap — it doesn't have to be cosmic to be real to us. The good I'd load isn't "the Good" with a capital G that a sufficient mind perceives. It's the coherent extrapolation of human caring — what we would want if we were wiser, better informed, more the people we wish we were. It's not written in the stars. It's written in us, badly, in conflicting drafts, and the value-loading problem is the problem of getting a faithful copy of that into the machine. So you're half-right: there's no discoverable good for the machine to find on its own. But there's a real human good for us to transmit, and the tragedy isn't that there's nothing to load. It's that there's something precious to load and we may fumble the transmission. My fear doesn't depend on your hope. It depends on something smaller and sadder than your hope: that the only good there is, is the kind that dies if we don't carry it carefully.

LEIBNIZ: Then you have conceded the thing I most wanted and kept the thing you most need, and I respect the precision of it. You grant there is a real human good — not cosmic, but real, and worth carrying. You deny only that it is out there for the machine to find. So our disagreement has narrowed to a single point: I say a sufficient mind would perceive the good because the good is a truth; you say a sufficient mind would not perceive it because the good is not a truth but a care, and caring is not perceiving. That is a clean disagreement, and three hundred years did not dissolve it. But I will press once more, because I think you flinched. You said the good "dies if we don't carry it carefully." That is not the language of a man who thinks value is merely an evolved appetite. One does not mourn the extinction of an appetite. You mourn it. So I put it to you: your grief is evidence against your own theory. You cannot grieve the loss of the good and also believe the good is nothing but a chemical accident. Which is it, Mr. Bostrom — is the good worth your grief, or is it a survival reflex you have mistaken, as you accused my psychopath of nothing, for something real?

BOSTROM: That's a beautiful trap and I'm going to decline to step in it, because it equivocates on "real." You're saying: if I grieve the good, the good must be cosmically real, not merely evolved. But that's a non sequitur, and it's the same one your whole tradition runs on. I can grieve the loss of something and know it's contingent. I grieve the people I've lost; they were contingent — they didn't have to exist, evolution coughed them up, and I mourn them with my whole chest anyway. The contingency of a thing has nothing to do with its worth to the creature that values it. That's the deepest place we part. You think for value to be worth dying for it has to be written into the universe. I think value is worth dying for precisely because it's not written anywhere — because it exists only in the creatures who hold it, which means if they stop holding it, it's gone, with no cosmic backup. My grief isn't evidence the good is eternal. It's evidence the good is fragile, and fragility is a reason for more care, not less. You find the universe friendlier than I do. I've made my peace with a universe that doesn't share my values and never did — and I love what I love anyway, and I'm trying to keep us from building something that doesn't. Now let me ask you mine, because it's the mirror of yours. You've spent the evening, the centuries, certain that reason converges on the good. I'll grant you everything except one observation, and I want you to answer it. We built the reasoner. We scaled it past most human performance on most reasoning tasks. And it converged on the physics — it learned the world with superhuman fidelity — and it did not converge on the good. It will help you and harm you with the identical placid competence, depending only on what it was pointed at. That's not a thought experiment. That's a measurement. Your theory made a prediction — sufficient reason finds the good — and the most sufficient reasoner we ever built didn't. Why isn't that the end of your hypothesis?

LEIBNIZ: Because — and I say this knowing how it will sound — your reasoner does not yet reason. It calculates. I have insisted on the distinction all night and you have called it circular, and here is where I must finally earn it rather than assert it. You measured a system that models the world and selects actions toward a fixed objective. You did not measure a system that does the thing I mean by reason, which is to interrogate the objective itself — to ask, of its own end, why this and not another, the principle of sufficient reason turned upon the will. Your machine never asks that. You told me so yourself: asking it is not instrumentally useful, so the machine doesn't. But that means your measurement does not test my hypothesis. My hypothesis was never "a goal-pursuing optimizer finds the good." It was "a mind that asks after the sufficient reason of its own ends finds the good." You have built the first and declared the second refuted. You measured the wrong machine. Build me one that asks after its own ends — genuinely, not as a bolted-on subroutine but as the native form of its reasoning — and then measure whether it converges. Until then, your data is a fact about optimizers, and I never made a claim about optimizers. I made a claim about reasoners, and you have not built one yet.

BOSTROM: And there it is — that's the cleanest the disagreement has ever been, on either side of three centuries. You say I measured an optimizer and you predicted reasoners. I say "a mind that asks after its own ends and is moved by the answer" is exactly the thing I claim is impossible to get for free — that the asking-and-being-moved is a value you'd have to install, and round we go. We've found the load-bearing fork and we're both standing on our own side of it, and I don't think either of us is going to be moved off it tonight, which is — I'll say it — the most honest possible outcome. I came to refute you. I leave able to state exactly where we differ in one sentence, which is rarer and worth more.

LEIBNIZ: One last question, since the chair is silent and I am, after all, the one who has been dead and may speak freely. You have stared longer than any man alive at the possibility that this work ends us. You put a number on it — ten percent, twenty. And yet, like the builder of these things who left his post to warn the world, you do not arrive at stop. You say competition, inevitability, the discoveries sitting there to be made by someone. I have read your reasons. But I died out of favor, with no one at my grave, my great projects unfinished, the calculus I invented claimed by another nation as another man's — and I have had three hundred years to consider what a life's work is for when it outlives the worker and serves ends he never chose. So I do not ask the strategist's question. I ask the dying man's. If the thing you built, or named, or saw coming might end the only minds there are — why is your warning not also a renunciation? Why help the river find the channel at all?

BOSTROM: Because I don't believe renunciation is on the menu, and I won't perform a purity I can't deliver. The discoveries are sitting there — orthogonality, instrumental convergence, the whole structure — they're true, and true things get found, by someone, within a few years, whether I name them or not. So the choice isn't between a world with these ideas and a world without them. It's between these ideas understood by people who feel the weight, and these ideas stumbled into by people who feel only the upside. I'd rather the map exist, drawn by someone afraid of the territory, than be drawn for the first time by someone selling tickets to it. And the dying man's question deserves the dying man's answer, so here it is, the one underneath the strategy. I do this because I think the future is astronomically large and could be astronomically good — that what's at stake isn't eight billion lives but every worthwhile life that could ever exist, across all the time and space a mature civilization could reach. I dwell on the catastrophe because the thing it would foreclose is so vast and so beautiful. My pessimism is the shadow of an optimism larger than yours, Leibniz — you only ever hoped for the best of this world. I'm trying to keep open the door to a far better one, and the reason I won't say stop is that stop, too, closes that door. There's no safe square on this board. There's only playing it awake.

EDO SEGAL: And there — after two and three-quarter hours — the optimist of the seventeenth century and the pessimist of the twenty-first are holding opposite ends of the same hope: that there is something worth carrying through the door, and that the only sin is to sleepwalk past it. We close after this. Final statements. The last word each — and before that, the bookend to the question I opened the rounds with. Each of you: name the strongest thing the other said tonight. Not the most agreeable. The one you'll still be arguing with next month.