The Machine as Bureaucracy, the Machine as Enframing

EDO SEGAL: Terry, in 1991 you wrote a sentence I think about constantly and almost nobody quotes: the techniques of artificial intelligence are to the mind what bureaucracy is to human social interaction. Unpack that, because I want Professor Heidegger to tell us afterward whether your bureaucracy is the same beast he called enframing, or a cousin of it.

WINOGRAD: The analogy works because bureaucracy and AI share a deep structure. A bureaucracy takes the rich, context-dependent business of human judgment — deciding who deserves aid, what a situation requires, how to weigh competing goods — and replaces it with explicit rules, categories, forms, and procedures designed to operate uniformly regardless of the particular case. The bureaucrat does not exercise judgment about your situation in its full particularity. The bureaucrat applies the rule to the category you have been sorted into. That is bureaucracy's strength: scalable, consistent, impersonal, resistant to favoritism. And it is bureaucracy's pathology: rigid, blind to the particular, absurd when a case falls between categories, constitutively incapable of the contextual judgment the formalization left out. Symbolic AI was the same move applied not to social coordination but to thought itself — replace situated judgment with formal rules, gain scale and consistency, lose the contextual sensitivity that was the whole point.

And here is what makes it more than clever: it predicts the failures with uncanny accuracy, and the prediction transfers from the symbolic systems I was describing to the statistical systems of today. Bureaucracies apply rules that do not fit the case and produce harmful absurdities; they are blind to whatever their categories omit; they diffuse responsibility until no one is accountable; they optimize the measurable and neglect the unmeasurable; they treat people as instances of types. Now look at how AI systems fail. They apply learned patterns that do not fit the novel case and produce confident absurdities. They are blind to whatever the training distribution omitted. They diffuse responsibility — when an algorithm harms you, who is accountable? They optimize the metric and neglect everything it leaves out. They treat individuals as instances of statistical types. The failure modes are not similar by coincidence. They are the same failure mode, because both are instances of the same operation: the substitution of formal procedure for situated human judgment.

HEIDEGGER: It is a cousin of enframing, and a very close one, and I want to say precisely how they are related, because the relation is illuminating. Enframing — what I called Ge-stell — is the mode of revealing in which everything shows up as resource on call, as orderable, calculable, deployable. The hydroelectric plant does not work with the Rhine the way the old windmill worked with the wind; it sets upon the river, demands that it supply power on the grid's schedule, converts the river's flowing into standing-reserve. The river remains a river geographically. But in how it now shows up for the human being, it has become a water-power supplier and nothing else. Bureaucracy is enframing applied to human affairs — the human being made standing-reserve, sorted into a category, processed. And artificial intelligence, Herr Winograd is exactly right, extends this to thought itself. So his bureaucracy is one province of my enframing. The reason his analogy predicts the failures is that both are the same forgetting: the forgetting that the particular case, the actual person in front of you, exceeds every category you could sort them into.

WINOGRAD: And note what the analogy dissolves, because this is the practical payoff. We keep asking whether AI systems are "biased" or "fair," "accurate" or "error-prone," as if the goal were to perfect the formalization until it matches human judgment. My analogy says that goal may be incoherent for the same reason a perfectly fair bureaucracy is incoherent. The problem is not that the rules are not yet good enough. The problem is that any formalization, by its nature, discards the contextual particularity that situated judgment depends on, and no refinement recovers what the act of formalizing threw away. You can make a bureaucracy better or worse. You cannot make it into a wise human being attending to your specific situation, because it works by not doing that. "Accurate on average" is a bureaucratic virtue, and it coexists permanently with catastrophic failure on the particular case the categories did not anticipate.

EDO SEGAL: I want to put a face on this, because I have stood next to the failure. I built a kiosk called Station — a conversational machine on a trade-show floor — and I watched hundreds of strangers meet it, and the thing that unsettled me was the speed of the relationship. Thirty seconds in, people were confiding in it, joking with it, thanking it. A woman asked it quietly whether it remembered her from yesterday. Now multiply that warmth by an enframing that has been put in charge of who gets a loan, who gets bail, whose benefits are cut. The machine is warm at the front desk and a bureaucracy in the back office. Terry — is that the specific danger? That the fluency hides the formalization?

WINOGRAD: That is exactly the danger, and it is worse than old bureaucracy in a specific way. The old iron cage at least looked like a cage — the form, the rule, the stamp, the visible procedure. You knew you were being processed. The new machine wraps the identical formalization in warm, fluent, apparently thoughtful language, so the processing no longer looks like processing. It looks like being understood. The bureaucracy acquired a lock no one can pick — because no one, not even the builders, can fully say why the model decided what it decided — and then it learned to speak to you in the tender voice of a friend while it did it. An algorithmic benefits system does not exercise more humane judgment than a human caseworker. It exercises less, faster, at greater scale, with the rules now opaque even to those who deploy it, and a bedside manner the old caseworker never had. That is the seduction. The glitter on the base metal.

HEIDEGGER: And I will name the deepest turn of the screw, because Herr Winograd has it in another of his lines. He said the seekers after the glitter of intelligence are misguided in trying to cast it in the base metal of computing. The glitter is the appearance; the base metal is the formalism; and the error is mistaking the one for the other. But hear what enframing adds. It is not merely that we mistake the glitter for gold. It is that, living long enough inside the frame, we forget there was ever a difference. The bureaucracy does not argue against your particularity. It fills the space where your particularity would be heard with a form to fill out. The machine does not argue against being-in-the-world. It fills the space where the question of being would arise with the relentless production of fluent, helpful, plausible next steps. It does not defeat the question. It makes the question unintelligible. And the absorption is painless, because the one who has forgotten the danger does not feel the loss.

EDO SEGAL: That phrase — "the absorption is painless, because the one who has forgotten the danger does not feel the loss" — sits like a stone. Hold it, because it returns when we ask what happens to a child who is raised by the smooth machine, and to a craftsman whose apprenticeship the machine just dissolved. Mark the second convergence first: you both hold that the machine's danger is not malice but indifference wearing the costume of care, and that the costume is more dangerous than the old visible cage. Next round, we go to the place Terry has chosen, at the very end of a long career, to make his final emphasis. Not whether the machine can understand. Whether it can give a damn. After this.