CONCEPT
The Dam Itself as Risk
Perrow’s most unsettling extension of normal accident theory: safety systems are themselves systems, subject to the same dynamics of interactive complexity and tight coupling that produce normal accidents in the primary systems they protect—and the interventions designed to build cognitive dams against AI-augmented risk can become the proximate cause of the failures they were designed to prevent.
On the night of April 25–26, 1986, the operators at Chernobyl were conducting a safety test. To verify that a protective mechanism would function during shutdown, they disabled the automatic shutdown systems that would have interfered with the test’s measurements. The safety test caused the disaster. This is the pattern that
Perrow’s framework identifies as the dam itself as risk: the safety measure, interacting with the primary system it was designed to protect, produces failure modes that neither the safety measure’s designers nor the primary system’s designers anticipated, because each designed against the other’s failures rather than against the interaction between them. In the context of AI-augmented work, the dams that
Edo Segal advocates—structured pauses,
AI Practice interventions, mandatory offline periods, sequenced workflows—are each subject to this dynamic. The mandatory break creates discontinuities and temporal compression