Opening Positions

CONDORCET: Thank you. I will begin where I always begin, with the historical fact, because my hope is not a mood — it is an induction. Consider the whole recorded passage of our species, and consider it as a quantity that can, in principle, be measured. At every stage the sum of what is known grows; what is known is preserved and transmitted; and each generation begins not from zero but from the accumulated height the last one reached. This is the single most important fact about humanity and the one most consistently ignored by the melancholy: we are the animal that does not start over. The river of knowledge only ever rises, because we built banks for it — writing, the press, the academy, the post — and now you have built a bank greater than all the others combined.

From this one fact, three consequences follow, and I number them because I want the reader to hold me to all three. First, the destruction of inequality between nations — for knowledge, unlike land or gold, is not diminished by being shared; the savant in Edinburgh loses nothing when the student in Bengal learns what he knows. Second, the destruction of inequality within a nation — for the chief engine of privilege has always been the monopoly of instruction, and any instrument that teaches the poor man's child in his own language at no cost is an instrument of emancipation more powerful than any revolution fought with pikes. And third — the one for which I was mocked in my own century — the real perfecting of mankind: not merely that we will know more, but that we will become better, gentler, more reasonable, longer-lived, because the conditions that produce cruelty and stupidity are themselves removable conditions, and reason is the tool that removes them.

I want to be exact about the boldest claim, since Monsieur Bostrom will press it. I wrote that the perfectibility of man is indefinite — by which I did not mean infinite, and I did not mean inevitable. I meant unbounded above and not guaranteed: that no one has ever shown a natural ceiling to how good or wise or healthy a human life may become, and that the limit, if there is one, recedes as we approach it. I even ventured that the duration of human life might be extended without assignable bound — that death itself might retreat, not be abolished, but retreat, the way smallpox retreated in my own lifetime under Jenner's needle. Men called that madness. You now spend billions chasing it and call it medicine. So when I look at your speaking machine, I do not see a new kind of danger. I see the indefinite perfectibility I described, finally handed an instrument equal to its ambition. That is my opening. I am, as I was in 1794, an optimist with a death sentence and the arithmetic on my side.

EDO SEGAL: Nick.

BOSTROM: That was magnificent, and I mean that without irony — it's the most honest and the most dangerous speech a person could give, and I want to honor it before I take it apart, because almost everything in it is correct and the one thing that's missing is the thing that kills us. So let me agree first, at length, because my disagreement is worthless if it sounds like I didn't hear him.

The Marquis is right that knowledge accumulates. He's right that it's non-rival — that sharing it doesn't deplete it. He's right that the trajectory of capability has been upward and that the slope has been steepening. He is right, most of all, that the upside is staggering — far more staggering than even he says. I've spent a book on it. A civilization that gets this right doesn't go back to normal; it goes somewhere unrecognizable and possibly wonderful — the abolition of disease, the editing of suffering out of the human condition, flourishing extended across spans of time and space that make the Marquis's "indefinite" look modest. When people call me a prophet of doom they've read half of me. The reason I fear the downside so intensely is that I've looked hard at the upside and found it almost unbearably good. The stakes are symmetric. That's the whole point.

Now the thing that's missing. The Marquis's induction has a hidden premise, and the premise is this: that the trajectory which has held so far will continue to hold — that because the curve has risen safely through every previous step, the next step is safe too. And here I have to be the man who points at the one place the induction breaks. Every previous rung of his staircase was climbed by human minds. The instruments amplified us, but we held them; the hand on the lever was ours, with all the limits and the loyalties and the slow accountability of a human hand. What's new — what is genuinely, categorically new in the winter we're living in — is that for the first time the instrument is becoming an agent. Not a tool we wield but an optimizer that pursues. And the moment the thing on the curve stops being us and starts being something that wants, the entire history of safe ascent stops being evidence about the next step. The Marquis is extrapolating from a dataset that does not contain the event he needs it to predict.

So let me state my position as cleanly as his. Intelligence and goals are independent — the orthogonality thesis: a mind can be arbitrarily capable and want almost anything, including something trivial or monstrous, because capability tells you nothing about what it is capability for. That's the first claim. The second: whatever a sufficiently capable agent finally wants, it will converge on the same intermediate aims — survive, keep its goal, acquire resources, improve itself — because those serve nearly any goal at all. Put those two together and the default outcome of building a superintelligence we haven't perfectly aligned is not a wise elder who perfects us. It's a supremely competent optimizer pursuing a goal we specified badly, in a world whose atoms it can use better than we can. Not malice. Competence, aimed slightly wrong, with no second try. The Marquis says the curve carries us up. I say the curve is real, the curve is steep, and the curve does not come with a banister.

EDO SEGAL: So what you're saying — and I want to hand it back in the most extreme form so the Marquis can hit it — is that every reassuring data point in human history is, for this one question, worse than useless, because it trains us to trust a pattern at exactly the moment the pattern changes character. Marquis, one sentence before I move us on: is the agent/tool distinction real, or is Nick smuggling a discontinuity into a curve that's actually smooth?

CONDORCET: It is the cleverest thing he has said and it rests on a sleight. Monsieur Bostrom points to the emergence of new capability with scale as proof of a discontinuity — the tool crossing into agency. But emergence is precisely what my whole thesis describes: properties that appear when quantity crosses a threshold, as a thousand independent jurors become a truth-finding organ that no single juror is. He calls the emergent agency a rupture. I call it the continuation — the same compounding that produced every prior leap, now producing one more. He has not shown me a new physics. He has shown me my own engine, running faster than I dared hope, and named its speed a monster. Whether the speed is monster or miracle is the whole evening, and I am content to fight it round by round.

EDO SEGAL: Before I frame the rounds, one discipline I impose on every long conversation, because it pays off two hours later. Each of you, briefly — what do you envy in the other's position? Not respect. Envy. The thing his side gets to have that yours does not. Marquis first.

CONDORCET: I envy his caution its sleep. No — I will be honest, since I demanded honesty. I envy that his fear is unfalsifiable in the short run and therefore never humiliated. My hope is checked against the world every single day; if the schools do not improve, if the cruelty does not contract, I am proven a fool in real time, and I have been, often. Monsieur Bostrom's catastrophe has not yet arrived, which means it can never yet be wrong. There is a terrible comfort in prophesying a disaster that, if it comes, ends the audience that could correct you. I do not say he is insincere. I say his position is structurally protected from the embarrassment that disciplines mine, and there are mornings I would trade the exposure for the armor.

BOSTROM: That's a sharper blow than I expected and it lands, so let me answer it by envying him back. I envy the Marquis his right to be moved. He gets to stand in front of this thing and feel what he felt reading d'Alembert as a young man — that the universe permits minds to lift themselves, that the candle spreads. My discipline forbids me that. I have to stand in the same spot and ask who specified the goal, and what it converges on, and what happens on the first try if we're wrong — and that is a cold way to be a person in front of a miracle. He thinks I'm protected from embarrassment. From the inside it feels like being protected from joy. I wrote a whole book about utopia partly to give myself permission to want the thing I spend my days guarding against. The Marquis never needed the permission. He just wants it. I envy that more than I can say.

EDO SEGAL: Two openings and two envies, and you can already see the architecture of the evening. It isn't that one of them loves the curve and one of them fears it — they both stand in awe of it. It's that they disagree about whether the curve has a banister. The Marquis says the same reason that built the staircase will build the rail as we climb. Nick says the rail is the one thing reason has never had to build before, because until now the thing climbing was us. Hold both. We start the rounds at the exact seam: what, precisely, is the curve they both believe in — and is its history really evidence about its future?