Marquis de Condorcet vs Nick Bostrom on AI · Ch12. The Crossing ← Ch11 Ch13 →
Txt Low Med High
HOUR THREE — THE CROSSING AND LAST WORDS
Chapter 12

The Crossing

Page 1 · The Crossing
Acceleration And Widening Gap
Acceleration And Widening Gap

EDO SEGAL: Here is how this round works, and then I disappear. For the next stretch you question each other directly. I won't restate, I won't rescue, I won't referee. The reader has heard me run the room for two and a half hours; now they need to hear what happens when I leave it. Marquis, you've conceded more to his fear than I expected. So you open. Ask him the thing you most want him to answer with me out of the way.

Accelerationism
Accelerationism

CONDORCET: Monsieur Bostrom. We agreed tonight that foresight is reason's answer to the unrecoverable error — that seeing the cliff early is the whole game. And then you said something that has not left me: that seeing the cliff and slowing the car are different muscles, and the second may be paralyzed. So I ask you plainly, with no host to soften it: do you actually believe we are doomed? Not "at risk." Doomed. Because if you do, your life's work is a strange thing — a man building maps for travelers he expects to drive off the road regardless. And if you do not, then your fear is, at bottom, a kind of hope wearing armor, and I would like you to take the armor off and tell me what you are really betting.

· · ·
Page 2 · The Crossing
Acceptance Theory Of Authority
Acceptance Theory Of Authority

BOSTROM: No. I don't believe we're doomed, and you've earned the unarmored answer, so here it is. My actual estimate is not "we die." It's something more uncomfortable than that: that the outcome is genuinely uncertain — and you of all people should appreciate why I refuse to collapse the uncertainty, because the honest aggregation of fallible forecasts is your invention. When researchers actually run the careful tournaments that try to forecast existential risk, the striking result is not consensus on doom. It is stubborn, structured disagreement — exactly the shape your jury theorem predicts when the evidence is genuinely thin. So my honest number is: the probability of catastrophe is high enough to be the most important fact about our era and low enough that everything we do still matters enormously. If I thought we were doomed, I'd stop. I haven't stopped. The reason I sound like the prophet of doom is a deliberate choice — in a world racing toward the cliff with the accelerator down, the marginal value of one more voice saying "faster, it's wonderful" is zero, and the marginal value of a voice saying "there is a cliff, and here is exactly where" is enormous. You sang the staircase because the starving needed to believe in one. I describe the abyss because the reckless need to believe in that. We are both, Marquis, saying the thing our moment is starved of. Yours was hope. Mine is the brake. Neither of us is reporting our whole heart. We're each supplying the missing half.

Access Points
Access Points

CONDORCET: Then let me ask the harder one, since you have been so honest. You have built a discipline, an institute, a vocabulary, a generation of frightened brilliant young people — all to apply the brake. And you said the brake fails because the race punishes whoever pulls it. So has your life's work failed by its own logic? Have you spent twenty years proving, with great rigor, that the rigor cannot save us?

· · ·
Page 3 · The Crossing
Access Vs Agency
Access Vs Agency

BOSTROM: [long pause] That's the question I ask myself at three in the morning, and I'll give you the answer I give myself. The race punishes the individual who brakes. It does not punish everyone braking together — that's the whole nature of a coordination problem; it's a trap precisely because the cooperative outcome is available and we can't reach it through individual action. So the only thing that can save us is the thing my work is for: changing what the actors collectively believe about the stakes, until braking together becomes thinkable. I haven't proven rigor can't save us. I've proven that rigor in one lab can't save us, and that only rigor shared can. My twenty years are a bet that you can move the whole convoy by changing the map every driver is reading. Which means — and I see you smiling, Marquis, you've trapped me — my entire project is, at bottom, an act of exactly your faith. I'm betting that collective human reason, given a true enough picture, can govern itself away from a cliff it would otherwise drive off. I built a cathedral of fear on a foundation of your hope. Now. My turn to ask you.

Access Vs Governance
Access Vs Governance

CONDORCET: Ask.

BOSTROM: You watched the Republic of reason — the Enlightenment made flesh, the thing you gave your life to — turn into the Terror and condemn you to die. You held the orthogonality thesis in your hands as a lived fact: a system optimizing hard for a stated good, virtue, that pursued it to mass murder with total internal consistency. You met my misaligned optimizer in person. It killed Lavoisier — the Republic has no need of savants. It's about to kill you. And you sat in that room and wrote that the mind progresses without limit toward the good. So I have to ask, with the host gone: is your optimism courage, or is it the most sophisticated denial in the history of philosophy? How do you write the staircase with the blade outside the door, unless some part of you simply could not bear to look at what you were actually seeing?

· · ·
Page 4 · The Crossing
Accidental Complexity
Accidental Complexity

CONDORCET: [long pause] You have asked the only question that has ever frightened me, and you have asked it better than my own conscience does. Let me answer it exactly. It was not denial. Denial would have been to write that the Republic was an aberration, that reason was innocent, that the Terror was a betrayal of the Enlightenment rather than one of its possible children. I wrote no such thing. I knew — I, who had voted in that Convention, who had watched the machine I helped build acquire its taste for blood — I knew that reason aggregated and armed and freed from the check of other reason becomes the most efficient cruelty there is. Your orthogonality thesis is not news to me, monsieur. It is my biography. And I wrote the staircase anyway, not because I could not see the abyss, but because I had decided what a man does with the abyss. He does not deny it. He builds the rail. The Sketch is not a denial of the Terror. It is the rail I built over it, in words, with the hours I had left — a deliberate act of construction against a darkness I saw more clearly than you ever will, because I saw it from inside the machine. You called it courage or denial. It is neither. It is work. It is the only work there is. And I would ask you to consider, Monsieur Bostrom, whether your fear, for all its rigor, is not sometimes the thing that excuses a man from doing it.

· · ·
Page 5 · The Crossing
Accompaniment Freire
Accompaniment Freire

BOSTROM: That lands. And I'll answer it with the truest thing I have. You're right that fear can be an excuse for paralysis, and I've watched it paralyze brilliant people in my own field — the despair that masquerades as realism. So let me say it plainly, to you, with no host: I reject the despair. I think we can do this. I think the rail can be built — not in words, in code and treaty and the slow construction of collective restraint — and I think it has to be built by people who can hold your hope and my fear in the same hand and not let go of either. You without me sing the convoy off the cliff. Me without you freeze the convoy in terror and the reckless drive around us. The thing that survives this floor is the two of us refusing to be only ourselves.

Accompaniment Not Manufacture
Accompaniment Not Manufacture

CONDORCET: Then for the first time tonight, monsieur, I have nothing to argue with. Which is, I think, exactly what our host was waiting to hear.

EDO SEGAL: It was. And I'm going to break my silence only to say: the reader just watched the optimist reveal he saw the abyss more clearly than the pessimist, and the pessimist reveal his whole life is an act of the optimist's faith. Hold that. We close after the break — final words, no winner, and then the staircase is yours.

· · ·
Continue · Chapter 13
Closing Statements
← Prev 0%
Ch12 Next →