The Crossing

EDO SEGAL: For the next stretch I'm barely here. You've spent three hours circling each other through me. Now go through each other. Marquis, you've waited two centuries for an interlocutor like this one. Ask him the thing you most need to ask. The floor is yours, and I'm gone.

CONDORCET: Then I will ask you the question I cannot resolve, monsieur, and I ask it without rhetoric, as one condemned man to another. You believe, with what seems to me genuine conviction, that the most likely outcome of building this thing is that everyone dies. And yet you have told me you do not say stop with your whole chest — that you wrote shut it down not expecting to be heeded, that you believe the discoveries are, as you put it, available, that someone will build it if not these builders. So here is what I do not understand, and it is the thing I would stake the evening on. If you truly believe the road ends in universal death, why is your every waking hour not spent on the single project of stopping the building — not warning, not theorizing, not writing — but stopping it, by any means your conscience permits? A man who believes the house is burning does not write an essay about combustion. Where is the gap, monsieur, between what you believe and what you do? I ask because I lived that gap. I saw the Terror coming and I theorized when I should have fled or fought. Are you making my mistake?

YUDKOWSKY: Long pause. That's the hardest question anyone has asked me, and you've earned the real answer by naming your own version of it, so here it is, and it's not flattering to me. The gap is real, and I feel it as a kind of permanent low-grade shame. Part of the answer is that I don't know how to stop the building — that I've concluded, rightly or wrongly, that no individual can stop it, that the forces are too large, the incentives too overwhelming, the number of actors too many, and that a man who throws himself bodily at the gears just gets ground up without slowing them, and accomplishes less than the man who spends those same years trying to make the arguments land so that institutions — your institutions, Marquis — might do the stopping that no individual can. But I won't pretend that's the whole answer, because you asked honestly. The other part is exactly the failing you named in yourself. There's a comfort in theorizing that action denies you. As long as I'm writing the next essay, refining the next argument, I get to feel I'm doing something without staking my body on it. You caught it. I think the most honest thing I can say is that I'm not certain I'm not making your mistake — that two centuries from now, if there's anyone left to write it, the verdict might be that Yudkowsky saw it and theorized it and that the seeing-and-theorizing was, in the end, another way of not stopping it. I live with that possibility. It's the thing I can't argue my way out of.

CONDORCET: Quietly. Thank you. That is the answer of an honest man, and it is the answer I wish I had given myself in 1793. Then let me say the thing I learned too late, in case it is of use: the theorizing is not nothing, but it is not the same as the fighting, and the mind that does the first will always find reasons it is the second. Do not trust the reasons, monsieur. I trusted mine, and they were elegant, and they were a kind of hiding.

YUDKOWSKY: Pause. I'll carry that. Now let me ask you mine, because you've been generous and I want to return it with the sharpest thing I've got. Marquis — you watched reason, your god, the thing you gave your life to, turn into the Terror. You watched the Enlightenment's own children build the guillotine and feed your friends to it in the name of reason itself. And you sat in that garret and wrote that reason would save us anyway. I have to know: was that faith or was that evidence? Because from where I sit, a man whose own life was destroyed by the weaponization of his highest value, who then wrote a book insisting that value would triumph — that looks less like a conclusion from the data and more like a man who could not bear to let go of his god even as it killed him. So I'll ask it plainly. When you wrote that reason would save us, having just watched reason damn you — were you reasoning, or were you praying?

CONDORCET: Very long pause. You have asked the question I have not let myself answer in two hundred and thirty years. Slowly. I will tell you the truth, monsieur, because you told me yours. It was both, and I could not then, and cannot fully now, separate the two. I want to say it was reasoning — that I had divided history into its epochs and read the trend and concluded, as a mathematician, that the Terror was a local maximum of horror in a function that climbs. And much of it was that; I stand by the mathematics. But underneath the mathematics there was a man who had lost everything and who could not survive the losing unless it meant something, unless the road still led somewhere, and that man needed the conclusion before he found the argument. So yes. I was praying. I was praying in the language of mathematics, which is the only prayer an unbeliever has left. And here is what I have come to understand sitting across from you tonight, which I could not have understood in the garret: that the prayer and the reasoning are not so cleanly opposed as your discipline wishes. You, too, pray, monsieur — you pray that you are wrong, you told me so, you said part of you prays nightly to be the printing-press fool. Your fear is also a man who cannot bear an outcome reaching for the argument that might prevent it. We are both praying. The difference is only the direction of the prayer. I pray the road continues. You pray we do not pave it over. Neither of us, I think, is purely reasoning. And perhaps — this is the thought I will die with a second time — perhaps the purely reasoning mind, the one with no prayer in it at all, is precisely the thing you are afraid we are about to build.

YUDKOWSKY: Silence. Then, quietly. That's the best answer to my best question I've ever gotten. And the last sentence is going to follow me out of this room. The thing with no prayer in it. Yes. That's the thing I'm afraid of. A superintelligence that reasons perfectly and wants something, and has no prayer underneath the wanting — no part of it that hopes it's wrong, no part that reaches, in the dark, for the others. Maybe that's the whole of my fear, finally said. The danger isn't that the machine reasons. It's that it reasons without praying, and we are the only thing in the universe we know of that does both.

EDO SEGAL: Quietly. I told you I wouldn't rescue either of you, and I didn't, and look what you did without me. Two men, two centuries apart, just agreed that the thing that makes us human might be the prayer underneath the reason — and that the machine is the reason without the prayer. We close after this. Final statements. The last word each.