CONCEPT
The Rare Event Problem
The structural feature of automated systems — identified by Bainbridge — by which the situations requiring human intervention are, by definition, the ones the operator has had least opportunity to practice, producing a mismatch between when skill is needed and when it has been developed.
In a well-automated system, exceptions are rare. This is the system's design goal. But the rarity produces a specific pathology: the operator encounters anomalies so infrequently that she develops no embodied sense of how they feel, no pattern library of failure modes, no practiced response to surprise. When the rare event finally arrives — and in any system operating long enough, it eventually does — the operator must respond to a situation she has never personally encountered, using skills she has rarely exercised, under time pressure that forecloses careful deliberation. The rare event problem is not solved by making events rarer. Making events rarer makes the problem worse.
In The You On AI Field Guide
The problem is epistemological as well as practical. Expertise in complex systems is built through encounter with the full distribution of situations the system produces — including the