Anomaly Detection (Klein)

In the AI Story

Darlene's detection of sepsis illustrates the architecture of anomaly detection. The infant's color was 'not right' — not the pink flush of a healthy infant but something she could only describe. The movement pattern had shifted slightly. None of these cues would have triggered alerts in monitoring systems. Together, they triggered recognition that activated immediate response. The detection was possible only because Darlene's pattern library contained thousands of hours of direct observation of what normal looked like in its thousand variations. Without the library, the anomaly was invisible.

Klein's framework identifies anomaly detection as the function AI systems most conspicuously lack. Large language models trained on pattern-consistent data produce pattern-consistent outputs. When the situation falls outside the training distribution, the model does not recognize the departure — it continues to generate outputs consistent with its patterns, which may be fluent, confident, and wrong. The asymmetry is structural: the expert knows when she does not know; the AI does not.

The capacity depends on something deeper than information. It requires embodied familiarity with the domain's specific rhythms, textures, and tolerances — the kind of knowledge that can only be built through direct practice. The expertise paradox is that AI-augmented work reduces the direct practice through which anomaly detection is built, even as it creates new forms of output that require anomaly detection to evaluate.

The stakes are sharpest in domains where failure is catastrophic. The AI coding tool produces output that compiles and passes tests, but contains a subtle security vulnerability that a well-populated pattern library would have flagged as 'not right.' The AI medical tool suggests a treatment consistent with presented symptoms but misses the subtle constellation of cues that would have suggested a rare differential diagnosis. In each case, the detection depends on the reviewer having built, through direct experience, a pattern library rich enough to register the deviation.

Origin

Klein identified anomaly detection through Critical Decision Method interviews that repeatedly surfaced moments when experts acted on feelings they could not articulate. 'The fire didn't look right.' 'Something about the baby was off.' 'The system was behaving strangely.' These reports were consistent across domains and correlated with accurate intervention in cases where colleagues without comparable experience had detected nothing wrong.

The concept connects to the broader literature on tacit knowledge and perceptual expertise, but Klein's contribution was demonstrating that the capacity operates in real time under field conditions and that it can be systematically studied through structured retrospective interviews.

Key Ideas

Deviation sensitivity. The richness of the expected pattern determines how subtle a deviation can be detected.

Gestalt registration. Anomalies are often registered as overall 'wrongness' before specific cues can be articulated.

Experience-dependent. The capacity builds only through direct engagement with the domain's normal variation.

AI's structural gap. Pattern-matching systems cannot detect deviations from patterns they do not represent.

Review limits. A reviewer without the pattern library cannot detect anomalies the library would have flagged — reviewing AI output is not equivalent to producing it.

Debates & Critiques

A recurring debate concerns whether sufficiently sophisticated AI confidence estimation can substitute for anomaly detection. Klein's position is that confidence calibration and anomaly detection are categorically different: confidence addresses uncertainty within the model's representation, while anomaly detection addresses situations the model's representation does not cover. The former can be improved incrementally; the latter requires representational resources the current architecture does not provide.