The Banality of Optimization

EDO SEGAL: Hannah, I want to put the most famous and most misunderstood thing you ever wrote at the center of the table, because Daniela is going to have to take a position on it whether she likes it or not. You went to Jerusalem in 1961 expecting radical, demonic evil, and you found — your words — a man "terribly and terrifyingly normal." Not a fanatic. A joiner. A clerk who could not think from any standpoint but his own and organized a genocide as a logistics problem. And you concluded that the gravest evil does not require monstrous motives — only thoughtlessness, distributed through a system, at scale. Tell me the part everyone gets wrong. And then, Daniela, before you attack it, I want you to do the harder thing: tell us what the banality of evil gets right about your machines.

ARENDT: What everyone gets wrong is that they hear it as an excuse, and it is the opposite of an excuse — it is a deeper indictment. To say evil can be banal is not to make it smaller. It is to make it more dangerous, because it means catastrophe does not need villains. If evil required monsters, we could guard against it by finding the monsters. But if it can be administered by ordinary, competent, unreflective people each doing their function and none confronting the whole, then the danger is everywhere there is a system in which people do their jobs without judging what their jobs accomplish. Eichmann's defense was that he was following orders, applying the rules, doing his duty as the system defined it. That defense was an abdication of judgment dressed as obedience. He had a rule, and he had the cases — human beings — and he applied the one to the others and declined to do the single thing that might have saved everyone: to judge, to bring his own faculty to bear on the actual reality of what he was doing.

RUS: I'll do the hard thing, because the experiment deserves it and dodging it would be cowardice. What the banality of evil gets right about my machines — exactly right, and I want it on the record — is the danger of the diffused, unaccountable system in which everyone attends to a part and no one owns the whole. That is a real failure mode of complex engineered systems, and it is not hypothetical. The training data was historical; no one chose its biases. The model was tuned by people who didn't design the deployment. The deployment was decided by people who didn't understand the model. The output was approved by an operator with half a second and a presumption that the system knows best. At every node, a functionary attends to a task; at no node does anyone confront the meaning of the whole. Hannah's word for that fog is exactly right, and any engineer who has shipped a real system in the world has felt it. The diffusion of responsibility is the most underrated risk in my field. So: steelman delivered, and I mean it.

Now the place it fails, and it is load-bearing. Eichmann's defect was not that he followed a rule. It was that the rule he followed was monstrous and he refused to judge it. Hannah collapses two completely different things into one word, "thoughtlessness," and the collapse is where her argument over-reaches. There is the thoughtlessness of the man who will not think about a deed he should refuse — that is moral failure, and it is everywhere, and it terrifies me too. And there is the thoughtlessness of a thermostat, which has no judgment to suppress and is doing exactly the narrow, well-specified, benign thing it was carefully built to do. A self-driving car braking for a child is not Eichmann declining to think about the child. It is the opposite — it is the carefully reasoned judgment of hundreds of people about how to protect children, encoded so it executes when the human can't. Hannah's horror at the system that "does not waver" assumes the wavering was the conscience. Sometimes the wavering is just the fatigue. You cannot indict a smoke detector for not agonizing.

ARENDT: A thermostat does not select a target. A thermostat does not decide who gets the loan, who is flagged as a risk, who the autonomous weapon kills. You have chosen the most innocent possible example — braking for a child, which any of us would endorse — and used it to launder the cases where the machine is making a genuine judgment about a human being with no judgment in it. The smoke detector is not the analogy. The analogy is the system that decides a man is a parole risk, or a benefits applicant is a fraud, and disposes of him as an instance of a pattern, with no one at any point confronting this man, this irreducible particular, this who and not what. That is not a thermostat. That is Eichmann's defect — computing when one should judge — built into the architecture and run a million times a day. And the operator with half a second who rubber-stamps it is not a safeguard. He is thoughtlessness given a human face and a signature. A loop that contains a human but excludes thinking is no improvement. It merely relocates the banality, and gives it an alibi.

RUS: And there I largely agree with you — which should surprise the audience. The benefits-fraud system, the parole-risk score, the human in the loop who can't actually re-decide: these are bad designs, and I will condemn them next to you. But notice what's happened. You've moved from "autonomy is the banality of evil" to "these specific deployments are the banality of evil," and that is exactly the move I want, because the second claim is one I can act on as an engineer. The first claim says: refuse the machine. The second says: design it so the human can confront the particular when the particular demands it; reserve the high-stakes judgment about persons for persons; let the machine carry only the load it is genuinely better at, in conditions where reliability is the moral good. That is not a rejection of autonomy. It is the responsible design of it. Your rule at the door — autonomy to do what, audited by whom — I accept it completely. I just reach a different conclusion when I apply it: not no, but not like that, like this.

EDO SEGAL: Let me name what just happened, because the reader can't see your faces. That was the first exchange tonight where neither of you was smiling — and it produced a convergence I want to mark and number. Mark this moment: convergence one. You both condemn, without reservation, the system that makes a consequential judgment about a particular human being while engineering out the possibility that anyone confronts that human. You disagree about whether that failure is intrinsic to autonomy, as Hannah holds, or a design choice that good engineering can refuse, as Daniela holds. That fork — intrinsic flaw versus bad deployment — is the live wire of the whole night. And it runs directly into the next round, which is about the word you both staked at the door. If the machine acts, who did it? Action without actors. After this.