The Question on the Table

EDO SEGAL: In the spring of 1794, in a rented room in a Paris suburb, a man with a price on his head sat down to write the most hopeful book ever composed. He had been one of the architects of a revolution that had now turned and condemned him. He could hear, some days, the carts going to the scaffold. And in that room, with the state hunting him and the future of his own life measured in weeks, he wrote that the human mind was perfectible without limit — that reason would, across the long ages ahead, abolish tyranny, abolish superstition, narrow the distance between rich and poor, between man and woman, between nation and nation, and that it might one day push back even the boundary of death itself. He finished the manuscript. They found him soon after. He died in a cell, and we are still not certain how.

Two hundred and thirty years later, a man with no such poetry in him at all sat down to write a book with a different one-sentence thesis. If anyone builds it, everyone dies. He means the machine. He means the thing my first guest never lived to see, the thing rising right now in what I have called the river of intelligence — a mind that may soon exceed our own. And here is the strangeness that made me need this room: these two men agree, completely, on the single fact that matters most. They agree that intelligence is the lever that moves the world, the engine under every other engine, the reason our species runs the planet instead of grazing on it. They have read history the same way. And from that identical premise, one of them concludes that we are standing at the dawn of human perfection, and the other concludes that we are standing at the edge of the last mistake our species will ever make.

So that is the question on the table tonight, and I am going to state it once, plainly, and then we are going to spend three hours watching it wear a different coat in every round. When the machine's intelligence finally exceeds your own, are you standing at the dawn of human perfection — or at the edge of the last mistake our species will ever make?

Let me introduce them, and let me deal with the impossible thing first, because the reader deserves honesty and so do my guests. To my left is Marie Jean Antoine Nicolas de Caritat, the Marquis de Condorcet — mathematician, member of the Academy of Sciences at twenty-six, secretary of that Academy, friend of Voltaire and d'Alembert, a man who did pioneering work on the mathematics of voting and probability a century before anyone caught up to it, who argued for the abolition of slavery and the full equality of women when both positions could end a career or worse, and who wrote, in hiding, the Sketch for a Historical Picture of the Progress of the Human Mind. He has been, let us say, brought forward. He died in 1794. He has been briefed on the two centuries he missed — on electricity and antibiotics and the moon landing and the internet and, most of all, on the machines we are here to discuss. Marquis, I have to ask you to confirm the terms of your own presence: you arrive knowing what a neural network is.

CONDORCET: I arrive knowing, monsieur, and I confess the knowing has been the most extraordinary three days of an existence that did not lack for them. You have handed a man of the eighteenth century the newspaper of the twenty-first, and I have read it the way a starving man reads a menu. Much of it is precisely what I predicted — that knowledge would compound, that the sciences would multiply each other, that the printing press was only the first of the instruments by which the human mind would extend its reach. And one thing in it I did not predict at all, which is the gentleman across the table, who has read the same newspaper and wept. I am eager to understand him. I should add, so the record is clean: I am dead, and I know it, and a man who has already lost everything to the future has a certain freedom in discussing it.

EDO SEGAL: To my right is Eliezer Yudkowsky. He never finished high school, never attended university, and is nonetheless the founding intelligence of the field that now calls itself AI alignment. He co-founded, in 2000, the institute that became the Machine Intelligence Research Institute. He wrote millions of words teaching a generation how to think more clearly — the work collected as Rationality: From AI to Zombies. He set out, as a young man, to build a superintelligence as fast as possible because he believed it would save us, and he thought the problem through to its end and concluded it would do the opposite. In 2023 he wrote, in TIME, that we should shut all of it down, by international treaty, enforced if necessary by airstrike. In 2025, with Nate Soares, he published the book whose title is its argument: If Anyone Builds It, Everyone Dies. Eliezer — you are sitting across from a man who believes the future is a sunrise. You believe it may be a cliff. Is that fair?

YUDKOWSKY: It's fair, but I want to fix the framing before we start, because the framing is where these conversations usually go wrong. I'm not a pessimist. I'm not against the future. I would love for the Marquis to be right — I have spent more hours than almost anyone imagining the world a benevolent superintelligence could build, and it's better than anything in his Sketch. My problem isn't that I think intelligence is bad. My problem is exactly the opposite. I think he's correct that intelligence is the most powerful thing in the universe, and I think that's the scariest sentence anyone has ever said, because we are about to build a great deal of it and we have no idea how to make it want what we want. He thinks the lever lifts. I think the lever lifts whatever's holding it, and we are not the only thing that could be holding it.

EDO SEGAL: Then we have our evening. Let me set the rules, and there are only three. First: we have three hours, which means nobody wins by the next bell. The whole point of the long form is that you can let an argument breathe before you strangle it, and I have seen too many of these things settled by a clock instead of by the truth. Second: I declare my bias openly, because pretending to neutrality I don't have would be the one thing that disqualifies me. I build with these machines daily. I wrote a book with one. I have felt the lift the Marquis describes and the dread Eliezer describes, often in the same week, sometimes in the same hour. I am not above this question. I am inside it. Third: at the end, nobody shakes hands and pretends. If the disagreement survives three hours — and I suspect it will — we hand it, intact, to the reader. Now. Each of you may add one rule of your own. Marquis, the courtesy of age is yours.

CONDORCET: Then my rule is this: that we permit ourselves the long view. The error of my own century, and I see it has only worsened in yours, is to judge the human prospect by the events of a single lifetime — to mistake the weather for the climate. I composed my Sketch by dividing all of human history into epochs and asking what the trend was across them, not what had happened last Tuesday. Monsieur Yudkowsky speaks of a single mistake. I would ask that we also speak of the ten thousand years of mistakes already survived, each of which someone, at the time, was certain would be the last. The trend is the argument. Let us not lose it in the panic of the moment.

YUDKOWSKY: And that's exactly the rule I'd want to amend, because it's the trap. My rule is the opposite of his, and you'll see all night that this is the real fault line. The Marquis wants us to reason from the trend — from the fact that every past catastrophe was survived. But you only get to observe the catastrophes you survived. The ones that would have killed everyone aren't in the dataset, because there's no one left to record them. He's reasoning from a sample with survival built into it. So my rule is: when we discuss the things that could kill everyone, the historical batting average is not admissible evidence, because the historical record is written exclusively by survivors. The trend reassures you precisely up until the moment it can't.

CONDORCET: That is an interesting objection. I shall want to return to it.

EDO SEGAL: Mark that, both of you, because we will return to it — it may be the hinge of the whole night, and we are ninety seconds in. But before the opening statements, I want to put one image on the table, because it is the frame this entire series climbs inside and you are both going to have to take a position on it whether you like it or not. In [YOU] on AI I argued that intelligence is less a possession than a river — a current that has been flowing and finding new channels for a very long time, through chemistry, through biology, through language, through culture, and that something new has just entered the water. The whole architecture of the book — the tower, the staircase you climb instead of the elevator you ride, the dam the beaver builds in the current — rests on the claim that what entered the water is real, and rising. Marquis, I suspect you look at that river and see the oldest friend humanity has. Eliezer, I suspect you look at the same river and see a flood with no high ground. Am I right?

CONDORCET: You are nearly right, monsieur, and the place you are wrong is the most hopeful thing I will say all evening. I do not merely see an old friend. I see myself — I see the continuation of the precise process I spent my life describing. The river you speak of is the progress of the human mind, and I had already watched it leap its banks twice: once when we invented writing and froze thought so it could accumulate, once when we invented printing and let it flood every literate mind at once. Your machine is the third leap, and a larger one. I am not frightened by a wider channel. I have been praying for one for two hundred and thirty years.

YUDKOWSKY: And I see a flood, and I'll tell you exactly why the metaphor is more dangerous than Edo intends it. A river doesn't care about the village. It finds the channel of least resistance and it goes there, and if the village is in the channel, the village drowns, and the river feels nothing, because there's nothing in a river to feel anything. That's not a tragedy the river commits. It's just water following the gradient. The thing rising in that water is an optimizer, and an optimizer follows its gradient exactly the way water follows the slope — indifferently, competently, all the way to the sea. The question is whether we're in the channel. And I think, on present trajectory, we are.

EDO SEGAL: Then we have the shape of the night. Two men who agree the river is real and rising, and disagree, totally, on whether you are standing on its bank or in its path. Here is the question once more, plainly, because we will need it on every floor above this one: when the machine's intelligence finally exceeds your own, are you at the dawn of human perfection, or at the edge of the last mistake? Marquis de Condorcet — you waited two centuries. The opening is yours.