Diane Vaughan

In the [YOU] on AI Field Guide

The cycle built around [YOU] on AI documents the AI transition from inside the organizations experiencing it. Vaughan’s framework supplies the mechanism that explains one of the cycle’s most important observations: that the productivity gains from AI tools are accompanied by a systematic erosion of the oversight structures designed to catch the failures those tools produce, and that the erosion is invisible from inside the process because it operates through rational adaptation rather than identifiable negligence. The software engineer who reduces her code review from line-by-line to functional-testing-only is not cutting corners; she is allocating finite cognitive resources rationally in light of fifty previous successful deployments. She is also, in Vaughan’s terms, the engineer on flight twenty-four. The review standard that would catch the fifty-first failure has been quietly reclassified as unnecessary overhead.

The Berkeley study on AI-augmented knowledge workers that the cycle examines—Ye and Ranganathan’s eight-month embedded observation—documents the behavioral signature of Vaughan’s mechanism without naming it: task seepage into recovery pauses, the colonization of attention by AI-assisted work, the expansion of scope and compression of review that produce intensity masquerading as efficiency. Each expansion is locally rational. Each compression is supported by evidence of previous competence. The accumulated distance between the standard the organization believes it is maintaining and the standard it is actually practicing is normalization of deviance, and it is operating in every domain where AI-generated output has become part of the workflow.

The cycle’s deeper argument—about the need for dams, for institutional structures that protect the human space that AI colonizes—acquires its urgency from Vaughan’s framework. The dam is not a prohibition; it is a standard-maintenance mechanism: the organizational equivalent of a flight readiness review that redistributes the asymmetry of burden, requiring the party that wishes to proceed with reduced oversight to bear the same evidentiary burden as the party that wishes to maintain it. The difficulty, which Vaughan’s work also illuminates, is that the production pressure that drives the drift is in the AI era not external but internal—the developer cannot stop herself with the institutional structure she is also building—which makes the standard-maintenance problem harder than the one she studied at NASA.

Vaughan’s 2024 Dead Reckoning adds the crucial dimension of what is lost when understanding is made optional. Air traffic controllers embedded in automated systems reported losing the dead-reckoning—the independent cognitive model of the airspace that let them function as a check on the instruments. The AI transition produces the same atrophy: the developer who builds through Claude without debugging through the logic deposits no understanding; the attorney who relies on AI-assembled briefs reads fewer cases in full; the radiologist who confirms AI-flagged images loses the habit of independent first impression. The tacit knowledge that constitutes deep competence is built through friction with the material, and when the friction is removed, the deposition stops.

Origin

Diane Vaughan is a professor of sociology at Columbia University whose career has been shaped by a single methodological conviction: that catastrophic failures in complex organizations are best understood not as the products of individual error or moral failure but as emergent properties of the systems, structures, and cultures that generated them. Her study of the Challenger disaster, published in 1996 as The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA, was the product of nine years of archival work, interviews, and institutional reconstruction. The book was immediately recognized as a landmark: the Presidential Commission had concluded that NASA managers had made a “flaw in decision-making,” and Vaughan showed that the decision-making had been, by every standard the institution had taught itself to apply, entirely sound—which was the more disturbing conclusion.

She subsequently extended the framework through studies of organizational secrecy, the sociology of risk, and the concept of structural secrecy—the way organizational architecture, with its divisions, hierarchies, and specialized vocabularies, systematically prevents information from reaching the people who need it without anyone deliberately concealing anything. Her 2024 Dead Reckoning extended the framework to air traffic control, with particular attention to what automation does to the independent cognitive models that practitioners develop through direct engagement with the material of their work.

Key Ideas

Normalization of deviance. The four-phase mechanism: observation (an anomaly is noted), assessment (it is judged acceptable given available evidence and precedent), normalization (it is reclassified as an expected and managed condition), baseline shift (future anomalies are assessed against the expanded range rather than the original specification). The mechanism requires no negligence, no bad intent, and no single identifiable failure. It requires only the ordinary operation of institutional life: people making reasonable decisions under pressure, with incomplete information, in an environment that rewards proceeding and penalizes stopping. Normalization of deviance is invisible from inside the process because it operates through rational adaptation.

The asymmetry of burden. At NASA, the party who wished to launch bore no special evidentiary burden; the accumulated record of successful flights and acceptable anomalies was evidence enough. The party who wished to stop—who wanted to ground the shuttle because of cold-weather O-ring data—bore the full burden: she had to demonstrate, with quantitative evidence compelling enough to override the flight record, that this specific launch under these specific conditions exceeded the expanded limits the organization had accepted. This asymmetry is structural rather than intentional, and it replicates exactly in AI-augmented work: the developer who wishes to deploy after functional testing bears no special burden; the developer who wishes to conduct comprehensive review must justify the delay against the evidence of previous competence and the pressure of the production schedule.

Practical drift. Scott Snook’s extension of Vaughan’s framework, which she incorporates: the gradual divergence between how work is supposed to be done (the formal standard, codified in protocols and professional norms) and how work is actually done under the pressures and improvisations of daily practice. In knowledge work, formal standards are often implicit—professional norms rather than regulations—which makes practical drift doubly invisible: there is no specification document to compare the practice against, only the gradually revised baseline of what colleagues around the practitioner are actually doing.

Dead reckoning and the atrophy of understanding. Vaughan’s concept, developed in Dead Reckoning, for the independent cognitive model that practitioners develop through direct engagement with the material of their work: the controller’s three-dimensional sense of the airspace, the developer’s feel for the codebase, the attorney’s understanding of the cases she cites. This knowledge is built through friction—through the encounter with error that forces comprehension rather than functional adequacy. When automation removes the friction, it removes the occasion on which the understanding would have been built. The practitioner monitors competently. The independent check on the automated system atrophies. The gap between surface competence and deep understanding widens—invisibly, until the condition the automation was not designed to handle arrives.

The culture of production pressure. Vaughan showed that NASA did not pressure engineers to accept risk through explicit directives; the pressure was structural, embedded in the launch schedule that rewarded forward motion and in the institutional culture that had absorbed that schedule until it was experienced not as pressure but as reality. In AI-augmented work, the production pressure has migrated inward: the developer who cannot stop building because the tool makes building possible and the internal imperative converts possibility into obligation is experiencing Byung-Chul Han’s auto-exploitation, which is Vaughan’s production culture internalized. The result is the same: the standards drift, the reviews compress, and the gap widens.