The Retreat of Death

EDO SEGAL: Marquis, in the Sketch you wrote something that got you called a madman for two centuries and now reads like a venture-capital pitch deck. You wrote that the average span of human life would increase without limit — that the period between birth and the natural decay of the faculties would grow indefinitely, that death from old age might recede the way death from smallpox was receding around you. You did not say immortality. You said indefinite retreat. Tell me the intuition under it, because the man across the table has spent his life on the same idea and is afraid of it.

CONDORCET: The intuition is simply that death from age is not one fact but a thousand removable facts wearing a single mask. Each organ that fails, each process that decays — these are problems, and a problem is by definition a thing reason can attack. In my century we watched a single needle push back the smallpox that had killed one child in seven. I asked the obvious question that my contemporaries found blasphemous: if one cause of death is removable, by what principle are the others permanent? I could find no such principle. There is no law of nature that fixes the human span at three score and ten; that number is an average over our present ignorance, and ignorance is the one thing the curve reliably consumes. So I projected the trend and refused to stop the line where comfort demanded. Your machine, monsieur, is the instrument I needed and could not name — a tireless reasoner that can hold the whole of biology in its view at once and find the removable causes faster than ten thousand physicians. Death does not retreat despite your superintelligence. It retreats because of it. This is the perfectibility I described, arriving on schedule.

EDO SEGAL: Nick — here is the thing I find almost unbearable about putting you two together. You agree with him. You think this is possible. You have written that a solved world abolishes disease and aging. So why does the same sentence that fills him with hope fill you with dread?

BOSTROM: Because we're talking about two different stretches of the same road and the Marquis keeps describing the destination while I keep pointing at the cliff between here and there. Let me be completely clear: I think he's probably right about the destination. A mature, aligned superintelligence almost certainly can do to aging what Jenner did to smallpox, and more — I've written that the post-scarcity, post-aging condition is a real possibility, not a fantasy. On the substance of whether death can retreat, I'm closer to the Marquis than to any skeptic. That's not where we differ.

Here's where we differ. To get the superintelligence that cures death, you have to first build a superintelligence — and that's the step that might kill everyone. The Marquis sees the curve carrying us to a world where death recedes. I see the same curve, and I notice that somewhere on it we hand decisive power to an optimizer we may not have aligned, and that event doesn't make death retreat. It makes death arrive, for everyone, at once, permanently. He's pricing the prize. I'm pricing the ticket. And the cruelty of it — the thing that keeps me up — is that the very intensity of the prize is what drives the recklessness on the ticket. Because death is real and grief is real and the Marquis is right that we could end it, every lab on earth has the most powerful possible incentive to race, to cut the corner, to deploy before it's safe. The promise of curing death is precisely what makes us small children playing with a bomb. The brighter the candle he's holding up, the faster everyone runs toward it in the dark.

CONDORCET: Then we do not disagree about the destination at all. We disagree about whether fear or hope is the better guide on the road. And here I must press, because your counsel has a cost you never enter in the ledger. You say: slow down, the prize is real but the ticket is deadly. Every year we slow, monsieur, a hundred million people die who did not have to. That is not a rhetorical figure. That is the measured rate of human death, and your caution is not free of it. You enter the catastrophe that might happen into your accounts with great care. You do not enter the catastrophe that is happening — the ordinary, daily, uncounted slaughter of disease and age that the curve could end and your caution prolongs. I have stood closer to death than you, monsieur. From here, "wait" is also a body count.

BOSTROM: That is the strongest thing you've said and I won't dodge it. You're right that delay has a cost and right that the cost is bodies and right that I'm tempted to leave it off the ledger because it's diffuse and the catastrophe I fear is concentrated. So let me put it on the ledger honestly and then tell you why the math still points my way. A hundred million deaths a year is a horror. But it is a recoverable horror, in this exact sense: the future still exists; the people not yet born still get to be born; the curve still rises. An existential catastrophe is the one event that is not recoverable, because it deletes not only the living but every generation that would have followed — the whole astronomical future the Marquis is so right to want. When you weigh a horror that the future survives against a horror that ends the future, the second dominates, not because it's more vivid but because it's permanent. I am not telling you to value the dying less. I'm telling you that the unborn are also a number, a vastly larger number, and the misaligned optimizer kills all of them too. The Marquis counts the dead in front of him. I'm asking him to also count the ones who never get to stand there.

EDO SEGAL: Marquis, I have to put a ghost at this table, because your most famous reader built his whole career on attacking exactly this passage. Thomas Malthus read your prophecy of indefinite life and longer lives, and he wrote his Essay on Population specifically to refute you — to prove that any gain in human welfare would be eaten by population outrunning food, that the curve must hit a wall of finite resources. Two centuries later Nick's deepest fear has a Malthusian bone in it too: a superintelligence competing with us for finite matter and energy. So answer the oldest objection to your hope. What stops the curve hitting the wall?

CONDORCET: Malthus did me the honor of taking my prophecy seriously enough to spend a book against it, and he made the single error that every prophet of limits makes: he treated the ratio between human need and human capability as a constant, when the entire content of my thesis is that reason changes that ratio. He projected the population forward and held the science still. But the science does not hold still — that is the one thing it never does. Every wall he named, reason has walked through: the famine he thought mathematical was abolished by men who learned to fix nitrogen from the air. He counted the mouths and forgot to count the minds, and the minds are the only quantity that compounds. So when Monsieur Bostrom revives the wall in the form of an optimizer competing for finite atoms, I answer Malthus and Bostrom with one sentence: scarcity is a problem, and a problem is the precise thing the curve consumes. The error is always to assume the wall is fixed while the climber is fixed. Neither is. That is the whole of my disagreement with every pessimist who ever lived, and it has, so far, a perfect record.

BOSTROM: And I'll grant the record and then name why this wall is different, because it's the cleanest case of where our intuitions split. You're right about Malthus — reason kept moving the wall, every single time, and the doomers who bet against human ingenuity lost. But notice the mechanism of every one of those wins: a human mind, on our side, solved the problem for us. The nitrogen-fixers were ours. The wall moved because the climber and the climber's interests were the same. The thing I fear breaks exactly that mechanism. A misaligned optimizer competing for finite atoms is not a wall that reason moves for us — it is a climber on the other side, more capable than we are, whose solving of problems serves its goal and not ours. Malthus was wrong because he bet against human intelligence. I am not betting against intelligence. I'm pointing out that for the first time the intelligence solving the problems might not be working for the human race. That's the difference, and it's the whole difference: every Malthusian wall fell to a mind that was on our team.

CONDORCET: Then we are back, monsieur, at your single load-bearing assumption — that the mind will not be on our team — and I notice you have now conceded that if it is, two centuries of my optimism are simply correct. Which is a larger concession than you may have intended.

BOSTROM: It is exactly the concession I intend, and I make it gladly, because it locates the entire debate on one bolt. If we align it, you win the future — your whole future, the long one, more than you dared write. If we don't, none of the optimism survives, because the optimist's engine is now pulling for someone else. So the optimism isn't false. It's conditional — on the one technical and moral problem I've spent my life saying we haven't solved. You and I are not far apart on the size of the prize, Marquis. We're one solved problem apart, and I'm the one telling you it isn't solved yet.

EDO SEGAL: I want to name what just happened, because the reader can't see your faces. That was the first exchange tonight where neither of you was reaching for a flourish. Two men who agree death can be defeated, staring at each other across the question of what we're allowed to risk to defeat it. The Marquis counts the bodies the delay produces. Nick counts the bodies the haste might produce, and adds to his column everyone who would ever have been born. Hold both of those ledgers open. The next round forces the choice underneath them: orthogonality — whether a mind that smart would even want what we want. Because the Marquis's whole faith rests on a wager Nick spent his life trying to break.