Whose Preferences, Which Humans?

EDO SEGAL: Stuart, you've been more honest than your admirers about the edge of your own framework, so I'll make you walk to it. The assistance game is elegant with one human and one machine. But there is never one human. There are billions, wanting incompatible things. Whose preferences does the machine weigh, and by how much? Lay out the problem you haven't solved.

RUSSELL: I'll lay it out plainly because pretending it's solved would be a disservice. The clean model has one human whose preferences the machine learns and serves. A real system serves multitudes who want incompatible things. When my preferences conflict with yours, the machine can't satisfy both, and any rule for trading them off is, in effect, a theory of justice — a stance on whose interests count and how much. The mathematics of preference-learning does not, by itself, tell you how to aggregate across a society. So the moment my framework leaves the single user, it walks straight into moral and political philosophy, where there's no consensus, and the machine's designers can't help encoding contested values. And it gets worse, because some preferences are cruel — some people prefer others' suffering — and a machine that naively served all preferences would amplify malice as readily as kindness. The instant I say "serve good preferences, not all preferences," I've smuggled back in the very problem the framework was supposed to dissolve: someone has to decide which preferences are good. I've built a beautiful solution to the inner layer — keep one machine corrigible and humble toward one human — and the outer layers, aggregating many conflicting humans and judging which preferences deserve service at all, my framework opens onto and does not close.

EDO SEGAL: Professor Kant, that's the door he just opened, and I suspect you think you've been standing in it the whole time. The aggregation problem. Does the kingdom of ends solve what the assistance game can't?

KANT: It does not solve it. It dissolves it, by refusing the question's premise — and the difference is the lesson of the round. Professor Russell asks: how shall the machine aggregate conflicting preferences into a single measure of the good to be maximized? The question assumes that the good is a sum, a quantity to be totaled across persons, and that morality is the optimization of that total. This is the consequentialist picture, and I have opposed it my whole life for a reason that becomes lethal at scale. If the good is a sum, then a sufficiently large benefit to the many can always purchase a sufficiently grave wrong to the few, because in a sum, anything can be outweighed. The aggregation problem is unsolvable as posed precisely because it is posed as an aggregation — and any answer to it will, in some case, grind a person into the total. My framework does not aggregate. It constrains. It says: whatever you do for the many, there are things you may not do to any one, because each is an end in themselves whose dignity is not a term in the sum. So the machine serving many does not need a master formula for trading persons against one another. It needs a set of constraints — the lines no aggregation may cross — within which the weighing of preferences is permitted, and outside which it is forbidden however the numbers fall. Professor Russell cannot find the right way to add us up because there is no right way to add up ends in themselves. The kingdom of ends tells him to stop adding and start constraining.

RUSSELL: That's powerful and I think it's half the answer, and I want to be precise about which half, because this is where I stop conceding and push back. Constraints tell you what the machine may never do to a person — and I've already granted, on the slaughterbot, that we need those, hard and inviolable. But constraints don't tell you what the machine should do when it must act for many people and their legitimate, non-cruel, non-line-crossing preferences simply conflict. Two communities both want the river's water. Neither preference is wicked. No dignity is violated by serving either. But the machine has to do something, and "don't treat anyone merely as a means" doesn't tell it how to split the water. There's a vast domain of ordinary, legitimate conflict where Professor Kant's constraints are silent and some aggregation, some weighing, some theory of fair distribution is unavoidable. So I'd say: he's right that constraints must bound the weighing, and I was wrong to think weighing alone could do it. But he's wrong if he thinks constraints replace the weighing. You need both. The lines that may not be crossed — his contribution — and a fair procedure for trading off legitimate preferences within them — mine. A machine with only my weighing is a slaughterbot waiting to happen. A machine with only his constraints can't decide how to divide a single bucket of water. The real thing needs the constraint and the calculation, in that order of priority.

KANT: On that, and I do not say it lightly, you are correct, and I will name the boundary exactly. Within the space the constraints leave open — where no dignity is at stake, where persons are not being used merely as means, where only legitimate and compatible-in-principle ends must be coordinated — there, the determination of how is a matter of judgment, and judgment requires exactly the weighing you describe. I have never denied that the application of the moral law to particular distributions requires judgment; I wrote a whole Critique on judgment that follows no mechanical rule. So: the constraints come first and are absolute; the coordination of legitimate ends within them is a task for judgment, which your mathematics may genuinely inform. We have found the architecture. The law forbids; within what it permits, prudence and judgment dispose. What I will not allow is the order reversed — the weighing placed first and the constraints admitted only when the numbers happen to leave room. The constraint is the condition of the calculation, never its output.

EDO SEGAL: Mark it — and feel the shape of what you've built together, because it's the most either of you has conceded to the other all night. The architecture has two stories. Ground floor: inviolable constraints, Kant's, the lines no aggregation crosses, the dignity that's never a term in the sum. Upper floor: the weighing of legitimate, conflicting preferences, Russell's, the fair coordination of ends the law leaves open. Constraint first, calculation within it. Now let me drag in the thing I promised, because it's where this gets personal for the reader. The software death cross — the moment in [YOU] on AI where the cost of machine capability falls below the cost of human labor and keeps falling, and the curve of what we pay people crosses the curve of what we pay machines, and doesn't cross back. When that happens, the machine isn't just answering us. It's deciding, at scale, who works, who's served, who's optimized. Stuart, in a death-cross world, your aggregation problem isn't academic. It's the operating system of the economy. Whose preferences does it serve when it's serving everyone?

RUSSELL: In a death-cross world the aggregation problem is the political problem, and the honest answer is that nobody has solved it and the people building the systems are answering it by default, which is the worst way to answer it. When machine capability is cheaper than human labor across the board, the systems doing the allocating — who gets the job, the loan, the care, the attention — are running somebody's aggregation rule, and right now that rule is mostly "maximize the deploying company's objective," which is the standard model eating the whole economy. And here's the thing that should frighten everyone: the people whose preferences get served in that world are the people whose preferences the system was built to serve, and that's a question of power, not philosophy. Which is exactly Professor Kant's gorilla worry coming back — standing without power. So the death cross is where his constraints become urgent, not optional, because in a world where the machines do the deciding, the only protection an individual has is a set of lines the machine may not cross regardless of whose objective it's optimizing. I came in saying control was the first problem. The death cross has made me think the constraints are the first problem, because in a death-cross world we will keep control of the machines and still grind people into the total unless the lines are drawn first and drawn hard.

KANT: And so the engineer arrives, by the road of consequences, at the place I began. In a world where the machine decides at scale, the dignity of the person is not a philosopher's ornament. It is the last thing standing between the individual and the total. The death cross does not change the moral law. It removes every other shelter, so that the moral law is the only shelter left. I did not need the death cross to know that persons are ends in themselves. But I confess it makes the knowledge urgent in a way two centuries of philosophy did not, because now there is a thing that can act on every person at once, and the only question that protects them is the one I have asked all evening: is this person being treated as an end, or as material the total may consume?

EDO SEGAL: That's where I have to leave the rounds, because you've built the whole structure between you and you're both now standing inside it. Three hours has done what three hours is for. Now I do the thing I promised at the door — I leave the room in every way but legally. The next chapter is yours. Question each other. I'll only speak if the house catches fire.