CONCEPT
Designing for the Exception
Bainbridge's prescriptive principle that automated systems should be designed around the conditions required for successful human intervention in rare events — not optimized solely for normal operation, with the human treated as an afterthought called upon when nothing else works.
Designing for the exception inverts the dominant approach to automation. Most systems are optimized for the common case: the routine flight, the standard transaction, the typical query. The human is designed into the system as a backstop for whatever the automation cannot handle. Bainbridge showed that this architecture guarantees the backstop will fail, because the conditions it is being asked to operate under — degraded skill, surprise, time pressure, poor situation awareness — are exactly the conditions in which human performance is worst. The alternative is to design the system so that the human's role in the exceptional case is actively protected: engagement is maintained during normal operation, skills are exercised rather than allowed to decay, situation awareness is preserved through active participation, and the transition from monitoring to acting is supported rather than punished. This is the principle behind high-reliability organizations, and it is the principle AI deployment systematically violates.
