AI as Bureaucracy

In a 1987 talk, Terry Winograd drew a striking structural parallel between artificial intelligence and bureaucracy. Bureaucracies achieve efficiency by formalizing processes—replacing situated judgment with rules, protocols, and standardized procedures. They work well within defined parameters. They fail at boundaries—where formal rules meet situations designers did not anticipate, where the human inside the bureaucratic structure knows the rule does not apply but cannot override it because the system does not recognize exceptions. AI operates identically: statistical patterns of linguistic appropriateness produce contextually competent outputs within the territory patterns cover, but fail when gaps between rhetorical pattern and substantive truth become consequential. The analogy illuminates the specific risk: not that machines produce obviously wrong outputs (those are easy to catch), but outputs wrong in ways that matter—substantively, conceptually, structurally—while being right in every easily-assessed dimension (grammatically, rhetorically, stylistically).