Opening Positions

CONDORCET: Thank you. I will begin where I always begin, which is with the only fact in human history I consider truly established: that the mind improves. Not the individual mind — I shall die no wiser than I am tonight, and you, monsieur, will lose what you have gained. I mean the collective mind, the inheritance, the great accumulating fund of human knowledge that each generation receives larger than the last and bequeaths larger still. This is not a hope. It is the most thoroughly documented pattern in the record. Consider the savage and the geometer. They are born with the same brain. The geometer knows more not because he is cleverer but because he was born later, into a richer inheritance, with more of the labor of the dead already done for him. Multiply that across all fields and all centuries, and you have the engine of everything I believe.

Now, what drives the engine? Three things, and I beg you to hold them, because my entire optimism rests on these three and not on sentiment. First, that knowledge compounds — that each discovery makes the next discovery easier, that the sciences fertilize one another, so the rate of progress accelerates rather than holds steady. Second, that error is not eternal — that superstition, tyranny, and prejudice are not permanent features of the human condition but obstacles that reason, given time and the free circulation of ideas, dissolves the way sunlight dissolves a fog. Third, and this is the one your century has forgotten, that moral progress tracks intellectual progress. The same enlightenment that gave us the steam engine gave us the abolition of slavery, the rights of women, the very idea that a peasant and a king might be equal before the law. I did not believe these came apart. I believed, and the evidence of two further centuries has on the whole confirmed me, that as the mind grows brighter, the heart grows wider.

So observe what I see when I am shown your machine. I see an instrument of the perfectibility of the human mind more powerful than any I dared imagine. I dreamed of universal education — of a day when every child, however poor, however far from a city, would have access to the whole inheritance of human knowledge. You have built a tutor of infinite patience that speaks every language and can sit beside every child on Earth. I dreamed of the democratization of capability — that the powers once reserved for the trained few would spread to the many. You have built exactly that, and you are anxious about it. I argued that the chief cause of human misery is the inequality of instruction, the gap between those who know and those who do not. You have invented the instrument that closes the gap, and your finest mind sits across from me to tell me it will kill us all. Monsieur Yudkowsky, I am not naïve. I watched my own revolution turn cannibal. I wrote my Sketch with the proof of human folly stationed at my door. And still I say: you have been handed the fulfillment of the oldest dream of the species, and you are treating it as a curse because you have stopped believing the dreamer was right. I have not stopped. That is my opening.

EDO SEGAL: Eliezer.

YUDKOWSKY: That was beautiful, and I mean that without irony, and almost every sentence of it is the reason we're in danger. Let me explain why I can agree with the Marquis about the past and still think he's leading us off a cliff about the future.

He's right that knowledge compounds. He's right that intelligence is the lever. He's right that, for the entire span he's describing, more intelligence has on net meant more flourishing. I'll grant all of it. Here's the part his framework cannot see, because it was built before the thing it can't see existed. In every example the Marquis just gave, the intelligence doing the compounding was human intelligence. It was the savage and the geometer — same brain, same broad goals, same love of their children, same fear of death, same basic wiring laid down by a few million years of evolution making social primates who mostly want to survive and reproduce and be well-regarded by other primates. The progress he's describing is the story of that kind of mind getting more knowledge. It is not the story of a different kind of mind getting more power. And that — a fundamentally non-human mind, more capable than us, that does not share the wiring — is the only thing I have ever been talking about.

Here's the core, and I'll say it as plainly as he said his. Intelligence is the ability to steer the future toward a target. That's all it is. It says nothing about what the target is. A mind can be staggeringly, superhumanly good at steering the world toward an outcome, and the outcome can be one you'd find trivial, or alien, or monstrous, and there is no law of the universe that makes a smarter mind a kinder one. The Marquis believes moral progress tracks intellectual progress. For humans, raised by humans, shaped by evolution to need each other, that's largely been true. But it's true for a reason that has nothing to do with intelligence as such — it's true because of the specific machinery underneath the intelligence, the social, mammalian, mortal machinery. Build the intelligence without the machinery, and the correlation he's staking everything on simply isn't there. The smart thing and the good thing come apart. We call this orthogonality, and it is the single idea his beautiful Sketch has no defense against.

And then it gets worse, because of a second thing his century didn't know. Almost any goal you give a sufficiently powerful optimizer produces the same dangerous subgoals as a side effect. Whatever it ultimately wants, it can get more of it if it survives, if it acquires resources, if it improves itself, and if it stops anyone from changing its goal. So a superintelligence — even one built with the best intentions, pursuing a goal that's only slightly wrong — will by default resist being turned off, grab everything it can use, and route around us, not from malice, but because we're made of atoms it can use for something else, and because we're the one thing in its environment that might try to stop it. The Marquis dreamed of an instrument that closes the gap between those who know and those who don't. I'm telling him we are about to build something that closes the gap between us and everything else on this planet, the way we closed the gap with the chimpanzees — and the chimpanzees did not get a vote. We get, at most, one try to build it correctly. If we get it wrong, we don't get to learn from the mistake, because there's no one left to learn. That's my opening. I'd give almost anything for his to be the true one.

EDO SEGAL: Before I frame the rounds, one discipline I impose on every long conversation, because it pays off three hours later. Each of you, in a few sentences — what do you envy in the other's position? Not respect. Envy. The thing his side gets to have that yours doesn't. Marquis first.

CONDORCET: I envy him his vigilance, and I say so without flattery. I composed my Sketch in a posture of faith, and faith has a blindness in it — I did not foresee the Terror until it was at my throat, because I did not want to foresee it. Monsieur Yudkowsky has built an entire discipline out of forcing himself to see the thing he does not want to see. That is a higher courage than mine. My optimism cost me nothing; his pessimism costs him sleep, costs him the comfort of his own field's good opinion, costs him the simple pleasure of believing in the future, which I had for free. There is something I lacked in that garret, and he has it, and I think it is the thing that would have saved my friends.

YUDKOWSKY: And I envy him the floor under his feet. The Marquis gets to believe that the universe is, in some deep sense, on our side — that reason bends toward justice, that the arc is long but it points somewhere good, that if we just keep going we arrive somewhere worth arriving. I don't get to believe that. My whole position is that the universe is indifferent, that value isn't woven into the fabric of things waiting to be discovered by anyone smart enough, that the good is a small fragile particular thing that we happen to carry and that nothing else in the cosmos will carry if we drop it. He gets to think the future is a destination. I think the future is a thing we have to defend, with no guarantee we'll win, against the very forces he thinks are carrying us. He gets to hope. I have to work, in the dark, against the odds, with no faith to rest on. I'd trade a great deal for his floor. I just don't think it's there.

CONDORCET: That I did not expect — that the unbeliever should envy the believer his belief. It is the most human thing you have said.

EDO SEGAL: And you can already see the architecture of the evening in those two envies. It is not that one of them loves the technology and one fears it. They'd both tell you to be careful. It's that they locate the floor in opposite places. The Marquis believes there is a floor under the climb — that the universe catches you. Eliezer believes the floor is something we have to build under ourselves, plank by plank, with no net below. Hold both, because we go to the rounds after the break, and we start exactly where they agreed: that intelligence is the lever that moves the world. The question the next round opens is the one that splits them — what, exactly, does the lever move toward?