The Question on the Table

EDO SEGAL: Somewhere in the world, in the time it takes me to say this, a few hundred thousand people are letting a machine choose for them. Not advise them — choose. The route it picks, they drive. The next song it queues, they hear. The reply it drafts, they send, lightly edited. A teenager in São Paulo accepts the video the feed selected because the feed has watched her for two years and knows, better than she does, what will keep her. A man my age, who should know better, asks the box whether he should take the job, and finds that its answer fits him so exactly he forgets to wonder where the fit came from. The machine has learned us. And the question nobody in those hundreds of thousands of small surrenders stops to ask, because the service is so smooth it feels like care, is the question we are here to spend three hours inside.

When the machine knows what you'd choose — when it can infer your wishes from everything it has watched you do — should it serve the law you ought to obey, or the wants it watched you reveal?

I have wanted to host this particular conversation for as long as I have understood it was the conversation. And I have done something that requires an apology, so let me make it at the door. One of my guests has been dead for two hundred and twenty years.

KANT: I am told this is no longer an obstacle.

EDO SEGAL: It is not, and I'm grateful you're a sport about it. For the reader: Immanuel Kant has been briefed on the present. He knows what a neural network is, what a large language model does, what it means that a system can predict your behavior. He has read the engineering. He reacts to all of it in character — which, as you'll discover, is more formidable than charitable. Professor Kant, I'll introduce you and you can correct me.

KANT: I will correct you regardless. Proceed.

EDO SEGAL: Immanuel Kant spent his entire life in one city, Königsberg, and from that single room reorganized the whole of Western philosophy. He asked not what the mind knows but what the mind contributes to knowing before any world arrives. He drew the boundary between what we can know — appearances — and what lies forever beyond us — things as they are in themselves. And in his moral philosophy he gave us the idea that may matter most tonight: that a person is an end in themselves, never merely a means, possessed of a dignity that has no price and admits no equivalent. He built the categorical imperative — the test of whether the principle of your action could be willed as a law for everyone. He is, by some distance, the most rigorous opponent of treating people as material that has ever lived.

KANT: That will do, though you have made it sound warmer than it is. The dignity of persons is not a sentiment. It is a constraint.

EDO SEGAL: Hold that word — constraint — because my other guest has spent his career trying to turn it into mathematics. Stuart Russell wrote the textbook the entire field of artificial intelligence learns from, the one on every shelf in fifteen hundred universities, and then turned around and told the field its foundation was cracked. He calls the mistake the standard model: we build machines as optimizers, hand them a fixed objective, and unleash all the capability we can give them upon it. He argues this is a structural error — that a sufficiently capable machine pursuing a fixed objective will pursue it past the point where we wanted it to stop. And he offers a fix that is, to my mind, the most adult idea anyone has had about this technology: build machines that are uncertain about what we want, that infer our preferences from our behavior, and that defer to us precisely because of that uncertainty. He calls it provably beneficial AI.

RUSSELL: I'll accept all of that, with one correction I suspect Professor Kant will enjoy. I don't say the machine should defer to us because deference is nice. I say it because deference is the rational policy for an agent that knows it doesn't know what we want. The humility isn't a virtue I bolted on. It falls out of the uncertainty.

KANT: We will see whether it falls out, or whether you have put it in and forgotten doing so.

EDO SEGAL: And there's the evening. Let me set the rules — there are three, and then either of you may add a fourth. First: we have three hours, which means nobody has to win by the next bell. The whole point of the long form is that you can let an argument breathe before you strangle it. Second: I press both of you, and I declare my bias at the door. I build with these systems daily. I wrote a book with one. And I have felt, late at night, the specific comfort of being served so precisely that I stopped asking whether I was being respected — which means I have skin on both sides of tonight's seam. Third: at the end, nobody shakes hands and pretends. If the disagreement survives three hours, we hand it, intact, to the reader. Professor Kant, a rule of your own?

KANT: One. No claim about what the machine should do may be admitted until we have asked by what right it is made — quid juris, not merely quid facti. That a system performs is a matter of fact. That its performance licenses a conclusion about what it knows, owes, or deserves is a matter of right, and right is not settled by performance. I will hold you both to it. I will hold myself to it.

RUSSELL: I can work with that. Let me add one too, then, because it's the mirror image. No claim about what the machine should do may be admitted unless you can say how a machine would actually compute it. A principle that can't be turned into an objective, or a constraint, or a procedure a system could follow is, for our purposes tonight, a wish. I don't mean it isn't true. I mean it isn't yet engineering.

KANT: We disagree already, and we have not begun. You believe the moral law is incomplete until it can be computed. I hold that the moral law is complete before any computation, and that what cannot be derived from reason cannot be repaired by it.

EDO SEGAL: You see why I wanted this. Before the openings, one image, because it's the frame the whole series climbs inside. In [YOU] on AI I argued that intelligence is less a possession than a river — a current that has been flowing and finding new channels through chemistry, biology, language, culture, for a very long time — and that in the winter of 2025 something new entered the water: a machine that learned to speak to us in our own tongue. The book's architecture, the tower and the staircase you climb instead of the elevator you ride, rests on the claim that what entered the water is real, and that the floor we're standing on now is the one where you decide what it's allowed to do to you. Stuart, you live in that water. Immanuel, I suspect you'll tell me the water was never the point.

RUSSELL: The water is real and rising, and the channel it found is wider than people think. But I'd reframe your floor. The question isn't only what the machine is allowed to do to you. It's whether you can stay in control of a thing more capable than you are. Everything else — dignity, meaning, the whole climb — is downstream of that. Lose control and the question of what it's allowed to do to you is answered by the machine, not by you.

KANT: And there is the inversion I came to resist. Professor Russell makes control the first question and dignity the second. I make dignity the first and control a consequence of having understood it. A machine kept under our power that still treats persons as material has solved his problem and not mine. The order is not a detail. It is the whole disagreement.

EDO SEGAL: Then we have our evening, and we have it on the first night either of you has spoken. Let me state the question once more, plainly, because every round we fight tonight is this question wearing a different coat. When the machine knows what you'd choose, should it serve the law you ought to obey, or the wants it watched you reveal? Professor Kant — you have waited two centuries. The floor is yours.