Structural Secrecy

In the AI Story

Vaughan developed the concept through her Challenger research, demonstrating that the O-ring data which should have grounded the launch was available within the responsible organizations but was progressively translated, as it moved from engineers to managers to decision-makers, from conditional technical assessment into categorical classification. The worry the engineers felt — the experience-based sense that the numbers were within limits but the limits were wrong — had no organizational channel through which to travel.

The concept extends Vaughan's earlier work on Controlling Unlawful Organizational Behavior (1983) and anticipates her later study Dead Reckoning (2024), in which she documented how air traffic controllers construct cognitive models of the airspace that extend beyond what the instruments display. Structural secrecy, in this expanded framework, is what happens when the organizational or technological architecture cannot transmit the kind of knowledge that lives in practitioners' nervous systems.

AI introduces a new dimension that Vaughan's original framework did not anticipate: technological opacity operating alongside organizational opacity. Large language models generate output through statistical processes distributed across billions of parameters, producing reasoning in a form that does not decompose into the inspectable chain of decisions a human reviewer could interrogate. This is not a flaw to be corrected in the next generation of models but a structural property of the architecture.

The interaction between organizational structural secrecy and technological structural secrecy compounds both. The reduced review depth that normalized deviance produces means reviewers are less likely to detect surface anomalies; the opacity of the model means they cannot detect reasoning-level anomalies even if they look. The combined effect weakens human oversight at both the behavioral and technological levels simultaneously.

Origin

The concept emerged from Vaughan's analysis of NASA's flight readiness review process, where engineering concerns expressed in technical language at Morton Thiokol were translated through organizational layers — each with its own vocabulary, standards of evidence, and interpretive framework — until the conditional, worried assessment of the engineers had become a categorical classification of acceptable risk by the time it reached the decision-makers.

Key Ideas

Information exists but does not reach. The relevant data is not suppressed; it is filtered, translated, and categorically reclassified as it moves through institutional channels.

Necessary filtering is the mechanism. Organizations must filter information to function; the filtering is structural, not malicious, and the critical signals lost are a byproduct of legitimate operation.

Translation between vocabularies. Specialized communities speak different technical languages, and translation between them sheds exactly the nuance that critical judgments depend on.

Tacit knowledge resists transmission. The experience-based sense that something is wrong — the expert's dead reckoning — has no organizational channel built for it.

AI adds technological opacity. Model reasoning is not inspectable in the form human review requires, extending structural secrecy from the organization into the tool itself.

Debates & Critiques

Scholars have debated whether structural secrecy is irreducible or whether sufficiently well-designed information architectures can eliminate it. Vaughan's position, reinforced by later work in high-reliability organizations, is that structural secrecy can be attenuated but not eliminated — organizations can build channels for tacit knowledge, cross-functional review, and independent verification, but the filtering inherent in institutional complexity will persist. The AI case intensifies the debate because technological opacity is not amenable to organizational reform.