The Line No Calculus Crosses

**EDO SEGAL:** Stuart, you've spent more than a decade on a fight that looks, on the surface, unlike the rest of your work — lethal autonomous weapons. Killer robots. Tell the reader what *Slaughterbots* was, and why this is the one issue where I've heard you reach for a word your mathematics can't give you: *never*.

**RUSSELL:** In 2017, with the Future of Life Institute, I helped make a short film called *Slaughterbots*. It shows swarms of small drones, each with facial recognition and a small shaped charge, hunting and killing designated people. We screened it at the United Nations. And its power was that it contained no science fiction — every component already exists. Lethal autonomous weapons don't need a breakthrough; they need only the integration of things we've already built, deployed at a scale that removes the human from the decision to kill. And that last phrase is the whole thing. I don't object to autonomy in machines as such. I object to delegating to a machine the decision to take a human life — letting a system select and engage human targets on its own judgment. Because it crosses a line, and because of the scaling: human soldiers are a brake on violence — wars need people willing to fight. Autonomous weapons remove the brake. A few people could deploy millions of them. They'd be weapons of mass destruction that are cheap, selective, and able to target by any trait a classifier can recognize. And here's where you've heard me strain, Edo, and you're right. When I argue this, I am reaching for an absolute prohibition. A bright line. *No machine may decide on its own to kill a person.* And my own framework — preference-satisfaction, weighing, probabilities — can only ever make that line very expensive. It can't make it absolute. I want a *never* here, and I've never been able to derive it from the math.

**EDO SEGAL:** Professor Kant. He's reaching across two centuries for your *never*. Can you hand it to him? And does the categorical imperative actually forbid the slaughterbot, cleanly, the way Stuart wishes his framework could?

**KANT:** It does, cleanly, and it costs nothing in probabilities, and I will show you exactly why, because this is the case where the difference between a law and a calculation is not academic — it is a count of the dead. Begin with the formula of humanity. The slaughterbot treats the human being it kills *merely* as a means — as a target, an instance of a classified pattern, a thing to be engaged — with no possibility whatever of respecting that person as an end. There is no rational consent the victim could give; there is no agency the machine acknowledges; there is only a pattern and a charge. To delegate the killing to such a system is to construct an apparatus whose entire operation consists in [treating persons as material to be processed](https://www.youonai.ai/fieldguide/med/instrumental_reason_eclipse). It is the formula of humanity violated in steel. And the violation does not depend on a single number — not on how many die, not on the probability of a targeting error, not on whether the war is just. It is wrong in its *form*: a thing deciding the death of an end in itself. Now test it the other way, by universalization. Could the maxim — *delegate to a machine the unsupervised decision to kill a human being* — be willed as a universal law, stated openly to all whom it governs? It cannot, because no rational being could will a world in which the decision to end a person is removed from any person who could be answerable for it, where one is killed by a process that no one chose against one in particular and no one can be called to account for. The maxim destroys the very accountability that makes a moral community possible. So: forbidden twice over, by both faces of the imperative, with no calculation admitted and none needed. *This*, Professor Russell, is what the *never* you have wanted all evening actually is. It is not a very large cost. It is the recognition that some acts are ruled out by the form of the act, before any cost is weighed, because they treat the person as the one thing a person may never be treated as — material.

**RUSSELL:** And I'm going to do something I rarely do, which is simply accept it. Yes. That's the argument I've wanted and couldn't build, and the reason I couldn't build it is that I started from preference-satisfaction, and preferences are quantities, and quantities can always be outweighed by other quantities, so I could never get to *never*. Professor Kant starts somewhere else — from a constraint on the *form* of the act that doesn't enter the ledger at all — and that's the only place *never* can come from. So here, on autonomous weapons, I think the deontologist is simply right and the consequentialist tools are simply the wrong tools. The decision to kill a human being is exactly the kind of decision that should be governed by an inviolable constraint, not an expected-value calculation, because the moment it's a calculation, someone with a big enough number on the other side wins it. I'll go further. I think this case is the *proof* of the broader concession I made earlier — that a learning machine needs hard deontic constraints under its optimization. Autonomous weapons are where you can *see* that a purely preference-weighing machine fails catastrophically, because it will always be possible to construct the scenario where the weighing comes out for the killing. The line has to be above the weighing. Kant's been telling me that all night. The slaughterbot is where I can no longer pretend he's wrong.

**EDO SEGAL:** I want to stop the room. That may be the largest convergence of the night, and it ran the opposite direction of where we started. We opened with Professor Russell saying control was the first question and dignity the second. On the killing machine, he's just said dignity — the formula of humanity, the inviolable line — is the *only* thing that does the work, and his own control framework can't reach it. Mark it. On the one case where the machine's act is the death of a person, the engineer concedes the philosopher's ground entirely: some lines are not costs, and the form of the act decides, before any number. Professor Kant — anything to add, or do you let the concession stand?

**KANT:** I let it stand, with one observation for the reader. Professor Russell says this case is the *proof* that a learning machine needs constraints above its optimization. I say it is not a special case at all. It is the ordinary structure of morality, made visible because the stakes are a corpse. Every act has a form before it has a cost. The slaughterbot only makes vivid what was always true: that the question "may I do this to a person" precedes the question "what will it yield," and that a civilization which answers only the second has already lost the thread, in every domain, not merely in war. The killing machine is loud. The quiet machines that treat persons as material to be optimized are doing the same thing in a register soft enough that no one screams. That is what I came two centuries to say.

**EDO SEGAL:** Hold that — the loud machine and the quiet ones, the same wrong at different volumes. The next round is the quiet one at its quietest and its largest: what happens when the machine has to serve not one of us but all of us at once, with our wishes in conflict — and where that collides with the death cross I keep promising we'll reach. After the break.