CONCEPT
The Field-Laboratory Gap
Klein’s structural diagnosis of why AI systems evaluated in controlled conditions fail in consequential practice—the distance between the idealized world in which a technology is trained and benchmarked and the irreducibly messy world in which it is deployed.
The laboratory is the world in which AI systems are trained, evaluated, and compared to human performance. Its conditions are controlled: tasks are well-defined, criteria are clear, feedback is immediate, and the set of possible situations is bounded by the experimental design. The field is the world in which AI systems are deployed. Its conditions are uncontrolled: goals shift mid-task, information is ambiguous, time pressure is real, consequences are irreversible, and the situation that has never been seen before is precisely the situation that is most likely to matter.
Gary Klein spent his career documenting this gap in the context of human expertise—demonstrating that laboratory decision science, with its optimal solutions and controlled variables, systematically failed to account for the adaptive intelligence experts deploy in natural settings. In the age of
large language models, the gap has not closed; it has widened, because the speed and scale of AI-assisted work mean that more decisions are made