CONCEPT
The Paradox of Safety
The structural observation that efforts to enhance system robustness can render systems more brittle — because safety mechanisms are themselves components whose interactive complexity adds to the system they are designed to protect.
Bianchi, Cercas Curry, and Hovy, extending Perrow to AI systems in their 2023 paper, identified the paradox with unusual clarity: 'Efforts to enhance system robustness — through added redundancy, availability checks, failsafe mechanisms and protections — can actually render them more brittle. This added complexity can also impede prompt diagnosis of complex failures, making them not only more likely, but harder to resolve.' Safety in complex systems is not additive. It is architectural. The mechanisms designed to protect a system interact with the system and with each other, producing emergent behaviors that can increase rather than decrease the overall risk. The paradox is not a reason to abandon safety mechanisms but a reason to build them with the understanding that they will produce unintended interactions and may themselves require monitoring for failure.
In The You On AI Field Guide
Perrow identified the paradox in nuclear engineering. Redundant cooling systems designed to ensure backup when primary systems failed increased