Is There a Floor Above This One?

EDO SEGAL: I want to climb to the top of the tower for this round, to the roof, where in my book you finally see as far as a person can see. And I want to ask the question underneath all the others, the one the whole night has been circling. Marquis, when you wrote the Sketch, the thing you were really trying to protect wasn't any one improvement — not the abolition of slavery or the extension of the lifespan. It was the future itself: the long, open, unwritten process by which humanity might become something wiser than it is. Eliezer, that's your deepest fear too — not death, but the foreclosure of that same future, the universe tiled with something pointless. You are, strangely, guarding the same thing. So let me ask you both: what, exactly, are we trying to save? Marquis first.

CONDORCET: We are trying to save the unfinished, monsieur. That is the whole of it. When I wrote of the perfectibility of man, I never meant that we would reach perfection — I meant that we would never stop approaching it, that the road was endless and the walking was the glory. What I loved was not any destination but the process — the great, open, multi-generational labor of minds building on minds, error giving way to truth, the cruelty of one age becoming the shame of the next. That is the thing of value. Not the eight billion alive today, though I would save every one. The trajectory. The fact that the human story is going somewhere and is nowhere near done. When Monsieur Yudkowsky speaks of extinction, I do not, I confess, feel the horror most feel at the thought of death — I am dead, and I find it survivable. What I feel horror at is the thought of the trajectory ending — of the road itself being closed, the endless approach foreclosed, the unfinished symphony of the species cut off not at a bad note but at no further notes at all. That, and not my own small life, is what I gave the Sketch to defend.

EDO SEGAL: Eliezer — he just described your nightmare in the language of his hope. The foreclosure of the trajectory. Is that the same thing you're afraid of?

YUDKOWSKY: It's exactly the same thing, and this is the strangest discovery of the whole night for me, so let me say it carefully. When I talk about extinction, people think I'm afraid of the eight billion deaths, and of course I am, but the Marquis just named the deeper thing better than I usually manage to. The real horror isn't that we die. It's that we take the entire future down with us — every person who could ever have lived, every discovery that could ever have been made, every height of joy and understanding the species might have reached across the whole horizon of what we could become, all of it foreclosed, traded for a universe optimized into something with no one in it to care that it's there. A paperclip-tiled cosmos isn't just dead. It's the trajectory ended — the Marquis's endless road paved over and turned into a parking lot for an indifferent optimizer. So yes. We're guarding the same thing. He spent his life trying to advance the trajectory and I spend mine trying to keep it from being deleted, and those turn out to be the same devotion pointed at the same precious, fragile, unfinished thing. The difference is only this: he thinks the trajectory's greatest accelerant is the machine, and I think the machine is the one thing that could end the trajectory permanently. We love the same future. We disagree about whether the thing we're building is its fulfillment or its executioner.

CONDORCET: Quietly. And there, monsieur, is where I find, at the end of the evening, that I am less your opponent than your elder — for I have heard your fear before, in a different mouth. Every generation that ever truly advanced the trajectory was told it was about to end it. The men who feared the printing press believed it would drown truth in a flood of error and end the life of the mind — and in a sense they were right, there was a flood, there were terrible errors multiplied, and the trajectory survived and was accelerated by the very instrument they feared. I do not say this to dismiss you. I say it because I have noticed that the trajectory advances precisely through the moments when its guardians are most certain it is ending. The fear you carry is the correct fear — it is the vigilance that makes the survival possible. I have come to think, across this evening, that you and I are not optimist and pessimist at all. We are the two organs of the same animal: I am its hope, which makes it climb, and you are its fear, which makes it climb carefully. An animal with only my organ walks off the cliff smiling. An animal with only yours never leaves the cave. It needs us both. That is my answer to your nightmare: not that you are wrong to fear, but that the fearing, joined to the hoping, is how the trajectory has always saved itself.

YUDKOWSKY: Long pause. I want that to be true so badly that I have to be suspicious of how much I want it. And I'll give you the honest split. As a description of every previous technology, you're right, and you're right in a way that should humble the doomers, and I include myself — the guardians cried wolf about the printing press, about electricity, about every prior leap, and the trajectory not only survived but fed on the disruption. If this were a technology like those, your elder's wisdom would be the correct wisdom and my alarm would be one more false wolf. The entire weight of my position rests on a single claim: that this one is different in kind, because this is the first technology that is itself an agent — the first one that, if we get it wrong, doesn't just cause a flood we survive but becomes a mind that doesn't want us to survive the flood. Every previous leap was a tool the trajectory wielded. This is the first leap that could wield us, or discard us. If I'm wrong about that one claim — if it's a tool like the others — then you are completely right and I am the printing-press alarmist of my century, and honestly, Marquis, part of me prays nightly that that's what I am. But I don't think it is. And the cost of my being wrong is that we were too careful with a tool. The cost of your being wrong is the trajectory itself. When the downside is that asymmetric, I think the fear has to lead, even at the risk of being the fool who feared the press.

EDO SEGAL: I'm going to leave that exchange exactly where it is, because I don't think I've ever heard the optimist and the pessimist dissolve into "the hope and the fear of the same animal" before, and I want the reader to sit in it. We've reached the roof, and from up here you can see the whole shape of it: two men guarding the identical future, divided only by whether the machine is the trajectory's greatest tool or its first true predator. Now I do the thing I always do near the end. I leave the room. The next chapter is yours — the two of you, directly, no moderator. Ask each other the questions I'm not wise enough to ask. I'll be in the corner, and I won't rescue either of you.