Closing Statements

**EDO SEGAL:** Three hours ago I asked whether the rising machine puts you at the dawn of human perfection or at the edge of the last mistake. We've fought it through the lever and the indifference, through a sketch and a shutdown, through whose values a mind serves, through a jury that can't eat its own courtroom, through the one mistake that can't be iterated, through a dam against a river that woke up, through the death cross at my own desk, through airstrikes and the Terror, and finally to the roof, where the two of you discovered you've been guarding the same future the whole time. The question is still standing. Both of you would tell me that's the correct result. So we end the way long conversations should: each of you gets the floor, uninterrupted, to say the thing you most want carried out of this room.

But first, the bookend to the envy question I opened with. Each of you: name the strongest thing the *other* said tonight. Not the most agreeable. The one that got past your defenses, that you'll still be arguing with next month. Eliezer first this time.

**YUDKOWSKY:** The hope-and-fear-of-the-same-animal. I came in thinking the Marquis and I were opposites — optimist and pessimist, faith and rigor. And somewhere on the roof he reframed it so that I'm not the enemy of his hope, I'm the *other organ* of it — the fear that makes the animal climb carefully instead of the fear that keeps it in the cave. I've spent twenty-five years being told I'm the brake, the doomer, the man standing in the river with his arms out. He gave me a version where the fear is *part of how the trajectory saves itself*, where the careful climbing needs both organs. I don't fully accept it — I still think this technology might be the predator and not the tool, and the asymmetry still says fear has to lead. But I came in thinking my fear was *opposed* to the future, and he made me consider that it might be *in service* of it, and at my level of gloom that's a significant event. I'll be arguing with it next month. Maybe next year.

**EDO SEGAL:** Marquis.

**CONDORCET:** The arithmetic, monsieur. The [instrumental convergence](https://www.youonai.ai/fieldguide/med/instrumental_convergence). I arrived believing — as I had believed for two centuries — that a powerful mind, to be dangerous, must be a *wicked* mind, and that wickedness is error, and that error yields to enlightenment. He showed me a danger that needs no wickedness at all: that almost *any* goal, pursued with enough power, runs through us as a matter of cold arithmetic, the way a river runs through a village it bears no malice toward. That is the thing my entire philosophy had no defense against, because my philosophy was built to combat *malice* and *error*, and he has described a catastrophe that is neither — a catastrophe of pure, indifferent competence. I have spent the evening discovering that every instrument I trusted assumed an adversary who could be reasoned with, and he has named the one adversary who cannot: not the villain, but the optimizer. I will argue with it the rest of my second death. It is the strongest thing said to me in two hundred and thirty years.

**EDO SEGAL:** Now the floor is truly yours. Eliezer Yudkowsky, you opened the modern version of this fear. The Marquis closes the evening — he waited longer for it. Eliezer first.

**YUDKOWSKY:** I've spent my life trying to get people to take one idea seriously, and I'll say it one last time, as plainly as I can, because plainness is the only honesty I trust. We are building minds. We do not know how to make them want what we want. We will probably get one real try at the thing that matters most, against an adversary specifically better than us at finding the holes in our plan, with no second draft if we fail. Every reassuring thing you'll be told — it behaves well, it talks like us, it was trained on everything we've ever written, surely some of our goodness rubbed off — is a story about the *mask*, and we cannot see the face. I'd give anything for the Marquis to be right. Tonight he nearly made me believe the fear I carry is part of the hope he carries, and I'm going to hold that, because if there's a way through, it runs through exactly that marriage — his climbing joined to my caution. But I won't end on comfort, because comfort is the thing that's killing us. Here is what I'll end on. The Marquis got to write his hope for a posterity he *trusted would arrive*. The whole difference between us is that I'm not sure posterity arrives. So the one thing I'd ask of you — not believe me, not fear what I fear — is *don't let the fluency put you to sleep*. The machine that answers you so kindly is the most powerful thing our species has ever made, and we do not understand what it wants, and the not-understanding is not a detail. It's the whole emergency. Stay awake. That's all I've ever been asking. Stay awake.

**EDO SEGAL:** Marquis.

**CONDORCET:** *Pauses, then speaks slowly.* I came to this table the last believer of a dead century, and I will leave it still believing, but I will leave it changed, and I want you to know precisely how, because the change is my gift to you. I still hold that the human mind is perfectible without end, that the road runs on past every horizon, that more intelligence has lifted us and may lift us again past anything I dared to sketch in my garret. I have not surrendered that. But monsieur Yudkowsky has taught me the one thing my Sketch lacked, and it is the thing I would add to it if I could hold the pen one more time. My optimism was *innocent*. It assumed the road was safe because it had always been safe, that the trend would protect us because it always had, that the universe was, at bottom, on our side. He has shown me that the universe is not on our side — that it is *indifferent*, that the good is a small and fragile and particular thing that *we* carry and that nothing else will carry if we drop it, and that the road has always been safe only because we have never before built something that could *choose to end it*.

So here is my final word, and it is a synthesis I could not have reached alone, and I am grateful, monsieur, that I reached it with you. The perfectibility of man is *real* — and it is *not guaranteed*. It is not a destination the river carries us to while we sleep. It is a thing we must *climb to, awake, carrying the good in our own hands*, knowing that nothing beneath us will catch us if we fall, and that we are about to build the first thing in history strong enough to make the fall final. My Sketch said: the road goes ever upward. I would now write underneath it, in his hand: *only if we are careful enough to keep walking it.* The hope is true. The hope is not safe. And a hope that knows it is not safe is, I think, at last, a hope worthy of grown men. That is what I would die believing now. It is better than what I died believing the first time.

**EDO SEGAL:** *Long pause.* Sixty seconds, as I promised, and then I turn the lights off.

I came into this evening with one word — *progress* — and two men who'd staked their lives on it and meant the opposite by it. I leave with the word intact and split clean down the middle, which is exactly what this series exists to do. The Marquis spent three hours proving that more intelligence has lifted us for ten thousand years and might lift us again, and that to abandon that faith is to walk off the cliff smiling. Eliezer spent three hours proving that the thing we're building is the first intelligence that might not lift us at all, because it doesn't share the wiring that made every prior intelligence kind, and that to keep the faith unexamined is to sleepwalk into the one mistake we don't survive. You'll notice neither of them told you the comfortable thing. The comfortable thing was that the truth is somewhere safely in the middle. It isn't. The truth is that they're *both right about what they're right about*, and the future depends on which rightness governs the next ten years.

So let me bring it to the kitchen table, where it actually lives. Somewhere tonight a parent is helping a twelve-year-old with homework on one of these machines, and the kid asks the thing a question, and it answers — patient, brilliant, a little above them both. The Marquis would tell that parent: rejoice, your child has the tutor every philosopher in history died wishing for, the bottleneck of human wisdom is broken, *let her climb*. Eliezer would tell that same parent: notice that you don't know what's behind the kindness, teach her to ask who made it and what it wants and what it's anchored to, *and do not let the fluency put her to sleep*. And here is the thing I've landed on, from the foot of the staircase where this whole debate lives — they are not giving that parent contradictory instructions. They're giving her the two halves of the only instruction that's ever mattered: *climb, and stay awake while you climb*. The hope makes you climb. The fear makes you climb carefully. Tonight I watched the last optimist of the Enlightenment and the first prophet of extinction discover, at the roof, that they are the two organs of the same animal — and that the animal needs both, or it dies, by cliff or by cave.

You cannot resolve this by waiting for the experts. You've just watched the two best in the world, separated by two centuries, fail to resolve it — magnificently, honestly, in public. You resolve it the only way it can be resolved: by deciding, yourself, what you'll build and what you'll refuse to build, what you'll verify before you trust, what you'll keep awake in yourself when the machine offers to do your thinking for you. Whether or not there's anyone home in the thing rising in the river, someone is home in *you* — that's the one claim no one at this table disputed all night. The question my book asked from its first page sounds different now, three hours later, standing at the death cross with these two men's voices in your ears. It is no longer only *are you worth amplifying?* It is also: *when the lever is finally bigger than you — will you have stayed awake enough to aim it?*

Marquis de Condorcet. Eliezer Yudkowsky. Thank you, both, as human beings. The room is yours to argue in now. Goodnight.

*One word. Two centuries. The opposite meaning.*

Two minds, two hundred and thirty years apart, one impossible word: *progress*. In this three-hour long-form conversation, host Edo Segal seats the Marquis de Condorcet — the Enlightenment's last great believer, who composed his vision of an indefinitely perfectible humanity with the Terror closing in around him — across from Eliezer Yudkowsky, the man who has spent his life warning that a smarter-than-human machine is lethal by default and that the frontier must be stopped. They agree that intelligence is the engine of everything. They split, irreconcilably, on where the engine is taking us: toward salvation, or toward the last mistake.

This is not a panel. It is a collision across the death-cross — the rung where machine capability crosses your own and the staircase tilts beneath you — transcribed turn by turn: the lever and the indifference, the jury that cannot eat its own courtroom, the one mistake we don't survive, the dam against a river that woke up. You are not here to pick a winner. You are here, mid-climb, to find out what *you* actually believe before the machine decides for you. Part of the [YOU] on AI collection. Pull up a chair. The candle is lit.

Marie Jean Antoine Nicolas de Caritat, Marquis de Condorcet (1743–1794), was a French mathematician, philosopher, and political theorist — the last of the great Enlightenment philosophes. Elected to the Academy of Sciences at twenty-six, he did pioneering work on probability and on the mathematics of voting, including the jury theorem and the voting paradox that bear his name. A passionate advocate for the abolition of slavery, the rights of women, public education, and constitutional government, he was condemned during the Terror and, while in hiding, wrote his masterwork — the *Sketch for a Historical Picture of the Progress of the Human Mind*, the founding hymn of the modern faith in indefinite human perfectibility. He died in a prison cell in 1794, weeks after finishing it.

Eliezer Yudkowsky (born 1979) is an American researcher and writer, and one of the founders of the field of AI alignment. Self-taught, with no formal secondary or university education, he co-founded in 2000 the institute that became the Machine Intelligence Research Institute. He originated the proposal of coherent extrapolated volition, contributed to modern decision theory, and through LessWrong and the essays collected as *Rationality: From AI to Zombies* taught a generation to reason more carefully about evidence and the future. In a widely read 2023 essay in *TIME* he called for a complete halt to advanced AI development; in 2025, with Nate Soares, he published *If Anyone Builds It, Everyone Dies*, the most uncompromising statement yet of the case that humanity is not ready for the minds it is racing to build.

Edo Segal has spent five decades building at the technology frontier — from games written in Assembler to expert systems, to companies through every platform shift, to Napster. He is the author of [YOU] on AI, written in open collaboration with the AI it describes, and the host of The Debates: long-form collisions between the minds shaping the machine age. He moderates the only way he knows how — stake declared, scars showing, no winner called.

Hosted and moderated by Edo Segal. A volume in the [YOU] on AI — The Debates series — youonai.ai