Small Failures and the Immune System

In the AI Story

The Citicorp Center crisis of 1978 illustrates the immune system operating as intended. William LeMessurier's tower, completed in 1977, was discovered a year later to be vulnerable to quartering winds — a load case he had not fully analyzed, combined with a construction change that substituted bolted for welded connections. The vulnerability was identified not by structural failure but by a Princeton student's question. The margin — measured in time between the vulnerability's identification and the arrival of a storm that would have exploited it — allowed remediation. Welders worked at night. The building stands. The small failure, caught in the window between error and catastrophe, never became a large one.

The Tacoma Narrows Bridge demonstrates the opposite case. Its deck was optimized to an extreme shallowness — eight feet deep for a 2,800-foot span. Under normal wind, it did not crack, deflect, or vibrate enough to register as a warning. The first manifestation of aerodynamic instability was also the last: oscillations grew without check because the margin in which they could have been detected at manageable amplitude had been consumed by the optimization. A deeper, less efficient deck would have oscillated earlier at lower amplitudes. The oscillation would have prompted investigation. The investigation might have led to remediation. The optimization eliminated all three steps by eliminating the initial oscillation altogether.

The analogy to biology is structural. The immune system does not prevent infection. It detects infection early — when the pathogen load is still small enough to be managed — and mounts a response that prevents systemic crisis. The system operates in the margin between initial incursion and point of no return. Remove the margin and the immune response has no window in which to function. The organism appears healthy until it does not, with no intermediate state.

Engineering's equivalent is the deflection that exceeds calculation by a small percentage, the vibration at an unpredicted frequency, the material fatigue slightly faster than test data suggested. Each is a departure from the design hypothesis, small enough to be observed without immediate consequence, consequential enough to signal that the real conditions are departing from the modeled ones. The engineer who observes and interprets these signals receives a second chance — the opportunity to update understanding before the departure becomes fatal. The optimized structure that does not produce these signals has no second chance. It operates within spec until it does not, and the transition is discontinuous.

Origin

Petroski developed the small-failures framework across To Engineer Is Human (1985) and subsequent work, drawing on his detailed study of structural failures and their warning signs. The immune-system analogy appears in varying forms across his writing, though Petroski typically preferred the engineering language of inspection, monitoring, and maintenance rather than the biological metaphor. The Henry Petroski — On AI simulation extends the analogy explicitly, arguing that AI optimization threatens the immune function itself — not through damaging structures but through producing structures that, by virtue of their optimization, cannot signal their own distress.

Key Ideas

Small failures are features, not defects. The crack, the deflection, the vibration are not design inadequacies to be eliminated. They are the structure's communication channel to the engineer, operating in the margin between normal function and catastrophic failure. Their absence is not a sign of superior design but potentially of insufficient margin.

The immune system requires margin. The warning signals occur in the space between specified capacity and actual failure capacity. If optimization reduces this space to zero, the signals have no space in which to occur. The structure becomes silent — and silence, in this case, is not a sign of health.

Time is the critical resource. The value of a small failure is the time it buys for intervention. A crack detected today that would become catastrophic in two years is valuable because the two years can be used. An optimized structure whose first failure is catastrophic provides no time and no option for intervention.

Efficiency and warning are in tension. The margin that enables early warning reads, to an optimization algorithm, as waste. Removing it produces efficiency gains. The gains are real. The cost — the elimination of the warning system — is invisible until the conditions arrive that the warning system would have detected.

Debates & Critiques

Defenders of optimization argue that modern sensor networks and continuous monitoring can replace the function of the structural factor of safety, detecting impending failure through data analysis even in structures optimized to the edge of their specifications. The argument has merit where sensors are comprehensive, reliable, and attached to response systems that can act within the time window available. Petroski's objection is that the argument moves the margin rather than eliminating the need for it: sensors and response systems themselves require factors of safety, the response window must be calibrated against the failure dynamics, and the entire apparatus depends on assumptions about failure modes that must be included in the monitoring design. The approach can work, but only when the monitoring system's own factor of safety is adequately conservative — which returns the argument to the original question of how much margin the engineer is willing to maintain against conditions she has not specified.