The Far Future and the Faces

EDO SEGAL: We're near the end, and I want to state the question on the table once more, because every round tonight has been it in a different coat. When we build AI, whose future are we answerable to — the people it's harming now, or the future it might end? And I want to come at it through the thing you, Nick, have made the foundation of everything: the astronomical stakes. The future could hold an unimaginable number of worthwhile lives, and an existential catastrophe forecloses all of them, so reducing that risk even slightly outweighs almost any present good. Timnit thinks that argument is a moral weapon pointed at the present. So I'm going to ask you both the hardest version. Nick — is the future real enough to outvote a face you can see?

BOSTROM: I think it has to be, and I think Timnit knows it has to be, because she just argued for it in the last round — she said the meaning of the moderator's struggle lies partly in the grandchildren it secures. That's a claim about future people having moral weight. The instant you say "we fight now so that they may live better," you've admitted the unborn count. My argument is only the refusal to stop counting at a convenient generation. The future is real in exactly the sense that your grandchildren are real to you before they're born — real enough to sacrifice for, real enough to constrain what you do today. I'm not asking anyone to value a hypothetical trillion over a visible one out of cold arithmetic. I'm asking them to notice that the line they've drawn around "the people who count" is drawn at the edge of what they can see, and that the edge of your vision is not a moral boundary. It's just a limit of attention.

EDO SEGAL: Timnit — he just turned your own last answer against you. You said the moderator's struggle gets meaning from the future it secures. Doesn't that concede Nick's foundation?

GEBRU: No, and the difference is the whole argument, so let me be exact. There's a world of difference between future people and the future as an aggregate to be optimized. I care about future people the way I care about present people — concretely, as people who will have faces, who will be harmed or helped by specific structures we build now. That's why I fight concentration: because the future people will live under whatever power arrangement we leave them. What I reject is the move where "the future" becomes one enormous number, detached from any actual person, that you can hold up to outweigh the visible — because the instant the future is a scalar instead of a people, it can be set to whatever size the argument needs, and it always comes out bigger than the worker. Nick says my moral boundary is just the edge of my attention. I say his is just the edge of his accountability — he's extended his circle of concern to a place so far out that nothing he does can ever be checked against a real outcome, which is very comfortable, because it means he can never be wrong about a person. I'd rather be answerable to a face than unaccountable to a trillion.

BOSTROM: That last sentence is the strongest thing anyone has said to me about my own work in a long time, and I'm not going to pretend it glanced off. "Unaccountable to a trillion." Yes — that's the disease the far-future view is prone to, and I've watched it infect people, the ones who became so fixated on astronomical stakes that they grew cavalier about a person in front of them, and I find them genuinely frightening. So let me concede the disease and then hand you back the part you can't dissolve. Even granting all of it — even accepting that the future-as-scalar is dangerous and corrupting and has been weaponized exactly as you say — the irreversibility doesn't go away. The reason I can't fully come to your side is not the trillion. It's the one-way door — the system that cannot be corrected once it is past us. Some mistakes you can be held accountable for because there's an after in which to answer. The specific catastrophe I fear is the one with no after — no survivors to subpoena anyone, no record, no second iteration, no court. My fear isn't really about the size of the future. It's about the permanence of the door. And permanence is not a scalar. It's a structural fact, and it's the one thing your accountability framework can't touch, because accountability requires a tomorrow in which to demand it.

GEBRU: And that — the door, not the trillion — is an argument I can actually respect, because it's not about bigness, it's about irreversibility, and irreversibility is something my framework takes seriously too. We don't deploy a face-recognition system into a justice apparatus and say "we'll fix it later," because the person it jailed doesn't get those years back. So I'll grant you irreversibility as a real category. What I won't grant is that your particular irreversible door is the one we should organize everything around, when there are dozens of irreversible doors slamming on real people right now — the surveillance that doesn't roll back, the concentration that doesn't un-concentrate, the craft knowledge that dies with the generation that wasn't trained. You've got one cosmic door. I've got a hallway full of them, with people in each one, and they're closing tonight.

EDO SEGAL: Stop — I want to mark that, because it's the cleanest the whole night has been. You just agreed on the category: irreversibility is the real moral weight, not size. And you disagree about which doors. Nick, one cosmic door with no after. Timnit, a hallway of present doors with faces in them. That's not a disagreement about values. It's a disagreement about a map. Hold it — because in a moment I'm going to leave the room, and let you two ask each other the question I can't ask for you. But first, the bookend I ask in every debate, and it's earned now. Name the strongest thing the other said tonight. Not the most agreeable — the one you'll still be arguing with next month. Timnit first.

GEBRU: The door. The irreversibility argument, stripped of the trillion. When Nick stops talking about astronomical numbers and talks about a specific kind of mistake that has no tomorrow in which to be corrected — that one I can't fully answer, and it'll keep me up. Because my whole framework runs on accountability, and accountability needs an after, and he found the one case where there might not be one.

BOSTROM: "Unaccountable to a trillion." The recognition that extending your circle of concern infinitely far can be a way of escaping accountability rather than expanding it — that a concern so large it can never be checked against a real outcome is a concern that can never be wrong, and that this is seductive precisely to people like me. I've used the size of the future as a shield without noticing. She named the shield. I'll be arguing with that one for years.

EDO SEGAL: Two strongest things, and notice they're the same shape from opposite sides — both of you found, in the other, the place where your own virtue becomes your own trap. That's what three hours buys. Now I leave the room. The Crossing — after this.