Gary Klein

Foreword

By Edo Segal

The thing I trusted most about myself turned out to be the thing I understood least.

I have spent my career making calls under pressure. Ship or don't ship. Hire or pass. Pivot now or hold the line. I never thought of these as decisions in any formal sense. I just — knew. The way you know which lane to merge into without running the calculation, the way you know a pitch is wrong before the client finishes their sentence. I wore that knowing like armor. It was fast, it was reliable, and I never once asked where it came from.

Then Claude arrived, and it could produce answers that looked exactly like mine. Same structure. Same confidence. Same fluency. And I could not always tell the difference between its output and what I would have produced myself. That shook something loose. Not doubt about the tool — doubt about my own knowing. If I could not explain how I arrived at a judgment, and the machine could arrive at something indistinguishable through a completely different process, then what exactly was my judgment made of?

Gary Klein spent forty years answering that question. Not about me specifically, but about fire commanders and nurses and military officers and chess masters — people who made extraordinary decisions under pressure and could not explain how. Klein did not dismiss their inability to explain. He built a science around it. He proved that expert intuition is not mystical. It is the compressed output of thousands of hours of direct engagement with a domain — pattern recognition so deep it operates below conscious thought, mental simulations so practiced they run in seconds, anomaly detection so refined it catches the thing the instruments miss.

That science matters now more than it has ever mattered, because AI produces the outputs of expertise without the process that builds expertise. And if we do not understand what the process actually is — what it deposits, how it operates, why it degrades — we will optimize it away and discover too late what we have lost.

Klein gave me the diagnostic framework I was missing. The language for what those ten minutes of formative struggle in Trivandrum actually built. The cognitive science behind why reviewing AI output is not the same as producing it yourself. The precise mechanism by which the next generation's judgment thins if we do not design for its development.

This book is that framework applied to the moment we are living through. It is another lens in the tower — one ground by a researcher who spent a lifetime making visible the invisible architecture of human competence. You will not look at your own expertise the same way after reading it.

— Edo Segal ^ Opus 4.6

About Gary Klein

Gary Klein (born 1944) is an American cognitive psychologist and a pioneer of the naturalistic decision-making movement, which studies how people make decisions in real-world conditions of time pressure, uncertainty, and high stakes. After receiving his Ph.D. in experimental psychology from the University of Pittsburgh in 1969, Klein founded Klein Associates, a research and consulting firm that conducted fieldwork with firefighters, military commanders, nurses, and other domain experts. His Recognition-Primed Decision (RPD) model, developed through thousands of critical incident interviews, demonstrated that experienced practitioners do not choose among options by rational comparison but by recognizing patterns and mentally simulating actions — a finding that challenged decades of classical decision theory. His major works include *Sources of Power: How People Make Decisions* (1998), *The Power of Intuition* (2003), *Streetlights and Shadows: Searching for the Keys to Adaptive Decision Making* (2009), and *Seeing What Others Don't: The Remarkable Ways We Gain Insights* (2013). His adversarial collaboration with Daniel Kahneman, published as "Conditions for Intuitive Expertise: A Failure to Disagree" (2009), established foundational criteria for when expert intuition can be trusted. Klein's work on the DARPA Explainable Artificial Intelligence (XAI) program and his ongoing research into human-AI interaction have made him one of the most important voices on the preservation of human expertise in an age of increasingly capable machines.

Chapter 1: The Fire Commander's Dilemma

In 1984, a cognitive psychologist named Gary Klein sat across from a fire commander in a small office in Cleveland, Ohio, and asked him a question that would reshape the science of human decision-making. Klein wanted to know how the commander decided what to do when he arrived at a burning building. The textbook answer, the one enshrined in decades of decision science research, was that the commander would generate multiple options, compare them against a set of criteria, evaluate the probabilities, and select the optimal course of action. This was the rational choice model, and it governed virtually every institutional framework for understanding how decisions should be made — in business schools, in military planning, in economics departments, in the emerging field of artificial intelligence.

The fire commander looked at Klein as though he had asked something slightly absurd. He did not generate multiple options. He did not compare alternatives. He arrived at the scene, read the situation, and knew what to do. The knowing was immediate, confident, and almost always correct. When Klein pressed him on how he knew, the commander could not fully explain it. He would say things like "the fire just didn't look right" or "I had a feeling we needed to get out." The explanations were maddeningly vague. The outcomes were maddeningly effective.

Klein spent the next four decades building a research program around what that fire commander was actually doing. The result was the Recognition-Primed Decision model, one of the most significant contributions to cognitive science in the twentieth century, and a framework that has become unexpectedly essential for understanding what happens when artificial intelligence enters domains that were previously the exclusive territory of human expertise.

The RPD model describes how experienced practitioners actually make decisions under time pressure, uncertainty, and high stakes — conditions that characterize firegrounds, intensive care units, military command posts, and, increasingly, the workplaces where humans collaborate with AI systems. Klein's research, conducted through thousands of field interviews with firefighters, nurses, military officers, chess masters, and neonatal intensive care specialists, revealed a pattern so consistent it demanded a new theoretical framework. Experts do not decide by comparing options. They decide by recognizing patterns.

The recognition operates below the threshold of conscious deliberation. A fire commander arrives at a one-story residential fire and registers, without articulating the process, a constellation of cues: the color of the smoke, the sound the fire makes, the feel of the floor, the behavior of the flames in the windows. These cues activate a pattern stored in memory — a pattern built through years of direct engagement with hundreds of previous fires. The pattern comes with an action script: a pre-packaged response that has worked in similar situations before. The commander does not choose among alternatives. The first option he recognizes is usually the one he implements. Klein's data showed that experienced commanders went with their first recognized option more than eighty percent of the time, and that this option was effective in the vast majority of cases.

The mechanism that makes this work is what Klein calls mental simulation. Before implementing the recognized action, the expert runs it forward in her mind, imagining how the situation will unfold if she takes that action. If the mental simulation reveals a problem — the action script does not fit some feature of the current situation — the expert modifies the action or cycles to a different recognized pattern. This is not optimization across alternatives. It is satisficing within recognition: finding an option that works rather than searching for the option that is best.

The entire process, recognition, simulation, action, takes seconds. From the outside, it looks like intuition. From the inside, it is the compressed output of extensive experience operating on rich environmental cues. Klein insists on this distinction because it matters enormously for understanding what AI can and cannot replicate. Expert intuition is not mystical. It is not irrational. It is not the opposite of analysis. It is analysis that has been internalized through practice until it operates faster than conscious thought — a form of rapid, parallel pattern-matching that draws on a library of thousands of cases stored in the expert's long-term memory.

The library is the critical element. It does not arrive with the job title. It is built, case by case, through direct engagement with the domain over years. Every fire the commander attends deposits patterns. Every patient the nurse treats refines the model. Every flight the pilot completes adds to the repertoire. The library is maintained through ongoing exposure — the patterns do not simply persist; they must be refreshed, updated, and corrected through continued practice. A fire commander who stops attending fires begins to lose the ability to read a structure. The patterns degrade. The recognition slows. The confidence erodes.

This is the foundation upon which Klein's analysis of artificial intelligence rests, and it is the foundation upon which the most important questions about the AI transition must be built. Because what happened in the winter of 2025, the arrival of large language models capable of producing expert-level output in natural language, represents the first time in history that the products of expert cognition can be generated without the process that built the cognition itself.

The AI system can produce the output — the code, the diagnosis, the brief, the analysis — without having built the pattern library through years of direct engagement with the domain. It arrives at the answer through a fundamentally different process: statistical inference across vast training sets, pattern-matching at a scale and speed that no human can match, but without the embodied, contextual, experientially grounded understanding that makes human expertise reliable in the situations where it matters most.

Klein's framework forces a distinction that the AI discourse has largely failed to make: the distinction between performance and competence. An AI system that produces expert-level output is performing at the level of an expert. This does not mean it possesses the competence of an expert. The expert's competence includes not just the ability to produce correct outputs but the ability to detect when outputs are wrong, to recognize anomalies that signal danger, to modify approaches when situations evolve in unexpected ways, to know the limits of her own knowledge. These capacities are built through the same experiential process that builds the pattern library. They cannot be separated from it.

Edo Segal captures the experiential dimension of this problem in The Orange Pill when he describes the engineer in Trivandrum who lost what he calls "the ten minutes of formative struggle." The engineer had spent four hours a day on mechanical plumbing work — dependency management, configuration files, connective tissue between components. Mixed into those four hours were perhaps ten minutes when something unexpected happened, something that forced the engineer to understand a connection between systems she had not previously grasped. When Claude took over the plumbing, the engineer lost both the tedium and the ten minutes. The tedium was worth losing. The ten minutes were not.

Klein's framework explains precisely why those ten minutes mattered. They were the moments when the pattern library was being updated. When the engineer encountered an unexpected configuration error, her recognition system flagged an anomaly — something that did not match the expected pattern. The anomaly triggered active sensemaking: the effortful, conscious process of figuring out why the situation deviated from expectation. The resolution of the anomaly deposited a new pattern, or refined an existing one, in the engineer's long-term memory. The next time a similar anomaly appeared, recognition would be faster, more confident, more accurate.

Remove the anomaly-generating experience, and the pattern library stops growing. Worse, existing patterns begin to degrade because they are no longer being refreshed through contact with the domain. The engineer can still evaluate AI output today because her patterns were built through years of hands-on work. But the next generation of engineers, the ones who never perform the hands-on work because the AI handles it, will lack the patterns needed to evaluate what the AI produces. They will be reviewers who cannot recognize the anomalies because they have never experienced the normal.

This is not a speculative concern. It is a prediction derived from four decades of research on how expertise develops and degrades. The pattern is consistent across every domain Klein has studied. The surgeon who stops operating loses the feel of tissue. The pilot who stops flying in manual mode loses the capacity for recovery when automation fails. The diagnostician who stops examining patients loses the ability to detect the subtle presentation that does not fit the textbook description. The expertise is not a permanent acquisition. It is a living system that must be fed by ongoing engagement with the domain.

The fire commander's dilemma, then, is not a historical curiosity. It is the defining challenge of the AI transition: How do organizations preserve the conditions under which human expertise develops when the technology that most needs expert oversight is simultaneously eliminating the experiences that build expertise?

The question has no clean answer. Klein's career has been built on the recognition that the messiness of real-world decision-making cannot be captured by clean models, that the laboratory paradigm of decision science, with its controlled variables and optimal solutions, fails to account for the adaptive intelligence that experts deploy in natural settings. The AI transition inherits this messiness and amplifies it. The fire commander who cannot explain how he knows the building is about to collapse is a model for the senior engineer who cannot explain how she knows the AI's output is subtly wrong. Both are drawing on pattern libraries built through years of direct engagement. Both are exercising a form of cognition that resists formalization. And both are essential precisely because the situations in which they are needed are the situations in which formal models, including AI models, are most likely to fail.

Klein's research offers one more concept that will recur throughout this analysis: the distinction between the world of the laboratory and the world of the field. In the laboratory, decisions are made under controlled conditions, with well-defined options, clear criteria, and known probabilities. In the field, decisions are made under time pressure, with ambiguous information, shifting goals, and consequences that are immediate and irreversible. The laboratory is the world in which AI systems are trained and evaluated. The field is the world in which they are deployed. And the gap between the two is where the most dangerous failures occur.

The fire commander did not decide in a laboratory. He decided in a burning building, with incomplete information, with lives at stake, with a clock running that would not pause for optimization. The AI system that assists him, or replaces him, will be evaluated on laboratory metrics — accuracy rates, response times, cost per decision. The metrics will show that the system performs well. They will not show whether the system can detect the anomaly that the metrics did not anticipate, the pattern that was not in the training data, the situation that the laboratory could not simulate because it had never been seen before.

Klein spent a career demonstrating that this gap is where expertise earns its keep. The gap between the expected and the actual, between the pattern and the anomaly, between what the model predicts and what the world delivers. The AI transition does not eliminate this gap. It widens it, because the speed and scale of AI-assisted work mean that more decisions are being made, faster, with less human engagement in the details that would allow anomaly detection.

The fire commander's dilemma is now everyone's dilemma. The question is not whether to use the tools. The tools are here, and they work, and the capabilities are real. The question is how to preserve the conditions under which the human capacities that complement those tools — pattern recognition, anomaly detection, mental simulation, sensemaking — continue to develop in the people who will need them most when the tools fail.

And the tools will fail. Not because they are poorly designed. Because the world is more variable than any training set can capture, and the situations that matter most are precisely the ones that have never been seen before.

---

Chapter 2: The Pattern Library and Its Enemies

A neonatal intensive care nurse named Darlene worked the night shift at a hospital in Dayton, Ohio. Klein's research team interviewed her as part of a study on how NICU nurses detect problems in premature infants — problems that often present with cues so subtle they do not register on the monitoring equipment.

Darlene described a moment that had occurred the previous week. She was making her rounds, checking on an infant who, by every available metric, was stable. Vital signs normal. Blood oxygen normal. No alarms sounding. The baby looked fine. But Darlene paused. Something was wrong. She could not say what. The baby's color was perhaps slightly different — not the pink flush of a healthy infant but something she could only describe as "not right." The baby's movement pattern had shifted, barely perceptibly, from the expected fidgeting to something more still. None of these cues would have triggered an alert in any monitoring system. Together, they triggered a recognition in Darlene that activated an immediate response: she called the physician, requested blood work, and the results revealed the early stages of sepsis. The intervention came hours before the monitoring equipment would have detected the problem. The early detection saved the infant's life.

Klein's analysis of incidents like this one revealed the cognitive architecture of expert anomaly detection. Darlene was not running a checklist. She was not comparing the infant's presentation against a decision tree. She was recognizing a departure from the pattern of normal that she had built through thousands of hours of direct observation. The pattern of normal was rich, multidimensional, and largely tacit — it included visual cues, behavioral patterns, temporal sequences, and contextual factors that Darlene could not fully articulate but could reliably detect. When the infant's presentation deviated from this pattern, even minimally, Darlene's recognition system fired. The deviation was the signal. The expertise was the capacity to detect it.

This capacity — the detection of meaningful deviations from expected patterns — is what Klein identifies as the most important human contribution in contexts where AI systems generate outputs that are statistically plausible but occasionally wrong. The AI produces output that conforms to the patterns in its training data. The human expert detects when conforming to the pattern is not the same as being correct. This detection depends on a form of knowledge that cannot be transferred through documentation, training manuals, or even direct instruction. It is embodied, experiential, built through thousands of hours of direct engagement with the domain, and maintained only through ongoing practice.

The pattern library is Klein's term for this accumulated repository of recognized situations and their associated features, actions, and expected outcomes. The library is not a database. It is an organic cognitive structure that grows through experience, is refined through feedback, and degrades through disuse. Its architecture reflects the specific history of the expert who built it — the fires she attended, the patients she treated, the systems she debugged, the failures she witnessed and learned from. No two experts have identical pattern libraries, because no two experts have identical experiential histories.

The richness of the pattern library determines the expert's capacity for three things that AI systems currently cannot replicate with the same reliability: anomaly detection, the recognition that something does not fit; mental simulation, the ability to imagine how a situation will unfold; and sensemaking, the process of constructing a coherent interpretation of an ambiguous situation. Each of these capacities depends on the library, and the library depends on the experiences that built it.

The enemies of the pattern library are anything that interrupts the experiential process through which it is built and maintained. Klein's research identifies several: prolonged absence from practice, the substitution of abstract training for direct engagement, organizational structures that prevent practitioners from receiving feedback on their decisions, and, increasingly, the automation of the tasks that provide the experiential raw material from which patterns are extracted.

The last of these enemies is the one that the AI transition has unleashed at an unprecedented scale. When AI systems handle the implementation work that previously provided the experiential foundation for expert cognition, the experiential flow is disrupted. The patterns that would have been built through hands-on engagement are never deposited. The anomalies that would have been encountered through direct practice are never experienced. The feedback loops that would have refined the library's contents are never activated.

Klein has been direct about this concern. In a 2026 podcast interview, he framed the question in terms that any practitioner would recognize: in an age of AI, what happens to the skills you never knew you were building? The question captures the insidious nature of the problem. The skills that are most at risk are not the ones that practitioners consciously value. They are the ones that develop as a byproduct of work that feels routine, mechanical, even tedious — the very work that AI is most eagerly automating.

Consider Klein's 2024 essay on exaggerated claims of AI superiority over experts. Klein examined a study that purported to show a machine learning algorithm outperforming emergency department physicians in predicting patient outcomes. The study was methodologically sophisticated and the results appeared clear. But Klein identified three problems that the study's design obscured.

First, what he calls a learning confound. The ML model was designed to learn from the data. The physicians never received comparable learning opportunities — they were evaluated on their existing knowledge without being given access to the data from which the model had learned. The comparison was structurally unfair, like testing a student who had studied the textbook against one who had not and concluding that the textbook was smarter than the student.

Second, what Klein calls smuggled expertise. The ML algorithm drew its predictive variables from the electronic health record. But the electronic health record is not raw data. It is the product of expert judgment — the observations, assessments, and diagnostic decisions of the physicians and staff who created the record. The algorithm was building on the expertise of the physicians it purported to surpass, not substituting for it. It was, in Klein's framing, parasitically dependent on the very expertise it claimed to outperform.

Third, what Klein calls big-data intimidation. The algorithm ultimately incorporated over sixteen thousand variables. But the empirical optimum turned out to require only two hundred twenty-four variables, and performance plateaued at approximately twenty. The massive data set was not producing proportionally massive insight. The marginal returns on additional variables were negligible beyond a threshold that was, by big-data standards, modest.

These three problems — learning confounds, smuggled expertise, and big-data intimidation — are not limited to a single study. They are structural features of the discourse that surrounds AI's relationship to human expertise. The discourse systematically compares AI performance at its best against human performance under constraints that the AI does not face, while simultaneously obscuring the degree to which AI performance depends on the human expertise it is being used to replace.

The Orange Pill cycle's core argument — that AI amplifies whatever signal it is given — takes on a more troubling dimension when viewed through Klein's lens. If the signal being amplified includes the accumulated expertise of the practitioners whose work generated the training data, then the amplification is not creating new intelligence. It is redistributing existing intelligence from the experts who generated it to the systems that extracted it, and doing so without the experiential mechanisms that allow the expertise to regenerate.

The extraction without regeneration is the dynamic that makes the pattern library's enemies so dangerous. The AI system trained on the accumulated output of expert practitioners can perform at an expert level today because the expertise is embedded in the training data. But if the deployment of the AI system eliminates the experiences through which the next generation of practitioners would build their own pattern libraries, then the expertise embedded in the training data becomes a finite resource being consumed without replacement.

Klein's NICU nurse Darlene illustrates the stakes. Her capacity to detect sepsis before the monitoring equipment was built through years of direct observation — nights spent watching infants, learning what normal looked like in its thousand variations, so that abnormal, even subtly abnormal, would trigger recognition. An AI system trained on NICU monitoring data might eventually learn to detect the physiological signatures of early sepsis. But the detection would depend on data that monitoring equipment can capture. Darlene's detection depended on data that monitoring equipment cannot capture — the subtle gestalt of an infant's appearance, the barely perceptible shift in movement pattern, the quality of color that a camera might register but that only an experienced human eye can interpret in the context of a specific infant's specific baseline.

If the next generation of NICU nurses is trained in an environment where AI handles the monitoring and alerts, those nurses will not develop the pattern library that Darlene built through years of unmediated observation. They will be competent operators of a monitoring system. They will not be the practitioners who detect the anomaly that the monitoring system cannot see. And when that anomaly occurs — the infant who is dying of something the system was not trained to detect — there will be no Darlene to pause and say, "Something is not right."

Klein frames this as a fundamental design problem for organizations adopting AI: the preservation of the conditions under which human pattern libraries are built and maintained. The question is not how to make AI more accurate, though that matters. The question is how to ensure that the human capacities that complement AI — the capacities that are needed precisely when AI fails — continue to develop in the people responsible for overseeing AI-generated outputs.

The answer cannot be found in the technology itself. It must be found in the design of the human systems that surround the technology: the training programs, the organizational structures, the workflow designs, the evaluation criteria that determine what practitioners do all day and therefore what experiential patterns they accumulate. If those systems are designed to maximize the efficiency of AI-assisted work — if every moment of a practitioner's day is optimized for throughput — then the formative experiences from which expertise develops will be squeezed out, and the pattern libraries of the next generation will be thinner, less populated, less capable of detecting the anomalies that matter most.

The pattern library's enemies have always existed. Klein documented them decades before the AI transition. But AI is the most powerful enemy the pattern library has ever faced, because it simultaneously eliminates the experiences that build the library and creates the conditions under which the library is most needed. The faster the AI generates output, the more decisions are made per unit of time, the more opportunities for anomalies to occur, and the fewer experiences the human overseers have had that would allow them to detect them.

This is not an argument against AI. It is an argument for a design philosophy that treats the preservation of human expertise as a first-order engineering requirement, not an afterthought. The fire commander's pattern library and Darlene's night-shift vigilance are not quaint remnants of a pre-AI world. They are the cognitive infrastructure upon which the reliability of AI-augmented systems ultimately depends. Designing systems that erode this infrastructure while depending on it is an engineering contradiction that no amount of computational power can resolve.

---

Chapter 3: Sensemaking Under Pressure

Klein distinguishes between two cognitive modes that experienced practitioners employ, and the distinction has become the central fault line of the AI transition. The first mode is pattern recognition — the rapid, largely automatic matching of current conditions to previously encountered situations. This is the mode the fire commander uses when he arrives at a structure fire and immediately knows the type of attack to mount. It is the mode that large language models approximate, at a different scale and through a different mechanism, when they generate outputs that match the statistical patterns of their training data.

The second mode is sensemaking — the effortful, conscious process of constructing a coherent interpretation of a situation that does not match any recognized pattern. Sensemaking is what happens when the fire commander arrives at a scene and something does not fit. The fire is behaving in a way he has not seen before. The smoke is the wrong color. The heat pattern does not correspond to the apparent location of the fire. In these moments, recognition fails — the current situation does not activate any stored pattern — and the expert must shift to a slower, more deliberate process of figuring out what is going on.

Sensemaking, in Klein's research, involves a set of cognitive operations that are fundamentally different from pattern matching. The expert generates a tentative frame — a candidate interpretation of the situation. She then tests the frame against the available evidence, looking for data that confirms or disconfirms it. She seeks additional information to resolve ambiguities. She modifies the frame when the evidence demands it, sometimes abandoning one interpretation entirely and constructing a new one. The process is iterative, context-dependent, and shaped by the expert's specific experiential history. Two experts confronting the same ambiguous situation may construct different frames, seek different information, and arrive at different interpretations — and both may be valid given the evidence available to each.

This is not a process that AI systems currently perform. Large language models generate outputs that are consistent with the patterns in their training data. When confronted with situations that fall outside those patterns — situations that are genuinely novel, that do not conform to any statistical regularity in the training set — the models do not recognize their own uncertainty. They do not shift from recognition to sensemaking. They continue to generate pattern-consistent outputs, outputs that may be fluent, confident, and wrong. Klein identified this as the deepest asymmetry between human expertise and AI performance: the expert knows when she does not know. The AI does not.

The knowing-that-you-don't-know is built through the same experiential process that builds the pattern library. Every time an expert encounters a situation that her patterns fail to match, she experiences the distinctive cognitive sensation of recognition failure — the moment when the expected pattern does not arrive and the expert is left in a state of active uncertainty. Over time, the expert develops a meta-cognitive sensitivity to this state. She learns to recognize recognition failure itself as a signal, a cue that the situation requires active sensemaking rather than routine response. This meta-cognitive capacity is, in Klein's framework, the hallmark of genuine expertise. The novice does not recognize recognition failure because she has too few patterns to distinguish matched from unmatched situations. The intermediate practitioner may recognize the failure but lack the sensemaking skills to resolve it. The expert recognizes the failure, activates sensemaking, and constructs an interpretation that guides effective action.

Klein's work on sensemaking was not conducted in the abstract. It was conducted in domains where sensemaking failures kill people. The Three Mile Island nuclear accident in 1979, the Mann Gulch fire disaster in 1949, the friendly fire incidents in military operations that Klein studied for the U.S. Army — in each case, the catastrophe resulted not from a failure of technical systems but from a failure of sensemaking. The operators at Three Mile Island had the data that would have revealed the actual state of the reactor, but they had constructed a frame — a sensemaking interpretation — that was wrong, and they assimilated incoming data into the wrong frame rather than recognizing that the frame itself needed to change. The smokejumpers at Mann Gulch had the perceptual cues that would have revealed the danger, but their established frame — routine fire, standard response — prevented them from recognizing the anomalies until it was too late.

In the AI context, sensemaking failures take a form that Klein's recent writing has addressed with increasing urgency. The AI system generates an output. The human operator reviews the output. The output is fluent, well-structured, and consistent with what the operator expects. The operator accepts the output and moves on. The acceptance is not the product of careful evaluation. It is the product of what Klein would call anchoring — the cognitive tendency to treat the first plausible interpretation as the correct one, especially when the interpretation is presented with confidence and fluency.

The fluency is the problem. Klein has noted that the most dangerous failure mode in AI-assisted work is not the obvious error — the hallucination that is clearly wrong, the output that contradicts known facts, the recommendation that is transparently absurd. The most dangerous failure mode is the subtle error embedded in otherwise correct output, the error that is plausible enough to evade detection by a reviewer who does not have the pattern library needed to distinguish correct from almost-correct.

Edo Segal documented this failure mode in the production of The Orange Pill itself. He describes a passage in which Claude drew a connection between Csikszentmihalyi's flow state and a concept attributed to Deleuze — something about "smooth space" as the terrain of creative freedom. The passage was elegant. It connected two threads beautifully. It sounded like insight. The philosophical reference was wrong in a way that would have been obvious to anyone who had actually read Deleuze, but that was invisible to a reviewer whose expertise lay in technology rather than continental philosophy.

Klein's framework provides the cognitive science behind this failure. Segal lacked the pattern library for continental philosophy that would have triggered anomaly detection when the Deleuze reference departed from the actual content of Deleuze's work. Without the relevant patterns, the passage did not produce the distinctive sensation of recognition failure. It produced, instead, the sensation of recognition — the feeling that the connection was apt, that the argument cohered, that the reference landed. The smoothness of the output defeated the detection process because the detection process depends on patterns that the reviewer did not possess.

This has implications that extend far beyond the production of a single book. In every domain where AI generates expert-level output for review by human operators, the reliability of the human oversight depends on the reviewer's pattern library being rich enough to detect the errors that the AI produces. If the reviewer's pattern library is thinner than the apparent expertise of the AI output — if the AI is producing work that exceeds the reviewer's capacity to evaluate it — then the oversight becomes nominal rather than substantive. The human is in the loop, but the loop is not providing the error-correction function that justifies the human's presence.

Klein's 2025 analysis of whether AI can perform his own pre-mortem technique illuminates a different dimension of the sensemaking problem. The pre-mortem is a method Klein developed in which a team imagines that a project has already failed and works backward to identify the causes of the failure. The method has been widely adopted in military planning, business strategy, and, increasingly, AI risk assessment. Klein acknowledges that large language models produce "surprisingly good" pre-mortem outputs — they can generate plausible failure scenarios, identify potential risks, and organize them coherently.

But Klein identifies what the AI pre-mortem misses. The in-person pre-mortem is not merely a technique for generating risk lists. It is a social process that reveals the team's internal dynamics — the unspoken concerns, the interpersonal tensions, the political undercurrents that shape how risks are perceived and addressed. A team member who has been reluctant to voice a concern about a powerful colleague's pet project may find, in the pre-mortem's hypothetical framing, the psychological safety to name the risk. A leader who observes which risks her team members identify gains information not just about the project's vulnerabilities but about her team's capabilities, blind spots, and confidence levels.

These social functions are, in Klein's assessment, as valuable as the risk identification itself. They are also functions that an AI pre-mortem cannot perform, because they depend on the embodied, interpersonal, politically situated reality of human beings working together in a specific organizational context. The AI produces the output. It does not produce the social process that the output is supposed to serve.

Klein's conclusion is characteristically precise: "One lesson here is how AI can devolve social tasks and coordination into data tabulations — and we may be poorer for it." The devolution he describes is not a failure of the technology. It is a feature of the technology — the capacity to extract the informational content of a process while discarding the relational content — applied to processes whose value lies substantially in the relational dimension.

The sensemaking that Klein has studied for decades operates at the intersection of the informational and the relational. The fire commander's interpretation of a burning building is informed by what he sees, but it is also informed by what his crew tells him, how they say it, what their body language communicates about their confidence level, and the institutional history of trust and credibility that shapes how he weighs their input. The nurse's detection of sepsis is informed by the infant's vital signs, but it is also informed by her relationship with the infant's specific baseline — the familiarity that comes from having cared for this particular baby over multiple shifts. The team's pre-mortem is informed by the analysis of project risks, but it is also informed by the interpersonal dynamics that determine which risks are named and which are suppressed.

In every case, the sensemaking that produces effective action depends on a form of knowledge that is simultaneously cognitive and social, simultaneously analytical and relational. AI systems can approximate the cognitive and analytical dimensions. They cannot approximate the social and relational ones. And in the domains that matter most — the domains where sensemaking failures produce the most serious consequences — the social and relational dimensions are not supplementary. They are constitutive. The interpretation is shaped by the relationship. The understanding is built through the interaction. Remove the interaction, and the understanding thins.

The AI transition is not merely a challenge to individual cognition. It is a challenge to the social infrastructure of sensemaking — the teams, the communities of practice, the mentoring relationships, the institutional cultures through which human beings collectively construct interpretations of ambiguous situations. Klein's research has documented this infrastructure in detail across dozens of high-stakes domains. The infrastructure is slow, expensive, and inefficient by every metric that the AI transition uses to measure progress. It is also the mechanism through which the human capacity for sensemaking — the capacity that is most needed when AI fails — is built, maintained, and transmitted across generations.

The preservation of this infrastructure is not a sentimental concern. It is an engineering requirement for any system that depends on human oversight for its reliability.

---

Chapter 4: The Expertise Paradox

In 2020, Gary Klein and his colleagues Robert Hoffman and Shane Mueller published the results of their work on the DARPA Explainable Artificial Intelligence program. The project's ambition was to address what DARPA leadership regarded as one of the most significant barriers to the effective deployment of AI systems: users did not understand how the systems worked, did not know when to trust them, and did not know how to detect when they failed. DARPA assembled eleven teams of AI researchers to build more explainable systems. Then, in a decision that reveals something important about the institutional landscape of AI development, DARPA established a separate team — Klein's team — consisting not of computer scientists but of cognitive psychologists, tasked with understanding what explanation actually means from the perspective of the humans who need it.

The assignment placed Klein at the center of a paradox that has only sharpened in the years since. The paradox is this: the AI systems that most need human oversight are the systems that are most difficult for humans to oversee. The more sophisticated the system, the more opaque its decision-making process, the less capable the human operator of detecting the system's errors, and the more the operator is forced to rely on trust rather than understanding. But trust without understanding is precisely the condition under which catastrophic failures occur, because the operator who trusts without understanding cannot distinguish between the situations where trust is warranted and the situations where it is not.

Klein approached the problem from the human side rather than the technical side. Most of the XAI teams were building AI systems designed to make AI more transparent — generating explanations of their own reasoning, highlighting the features that drove their predictions, producing confidence scores that were supposed to calibrate the user's trust. Klein's team asked a different question: what does the user need in order to form an accurate mental model of how the system works? Not what the system can tell the user, but what the user needs to know.

The distinction matters because explanation and understanding are not the same thing. A system can generate an explanation that is technically accurate and cognitively useless — that tells the user which variables contributed to the prediction without helping the user understand why those variables matter, or how the system would behave if the situation changed, or what kinds of errors the system is prone to. Klein's research with firefighters and military commanders had demonstrated that experts do not understand complex systems by receiving explanations of their components. They understand them by building mental models — internal representations of how the system works that allow them to predict its behavior, anticipate its failures, and recognize when its outputs deviate from what the model predicts.

The tool Klein's team developed was called AIQ — Artificial Intelligence Quotient. The name was deliberately chosen to shift the focus from the intelligence of the artificial system to the intelligence of the human user. Klein described the goal plainly: "We want to raise their IQ about the AI systems they're wrestling with." The toolkit included a set of non-algorithmic assessment instruments designed to help users identify the boundaries of their AI systems' competence — the conditions under which the system performs well and the conditions under which it fails.

The AIQ project revealed a pattern that Klein has observed across every domain where automation has been introduced: the people who are best positioned to oversee AI systems are the people who developed their expertise before the AI arrived. They have the pattern libraries, the anomaly detection capabilities, the mental models of how the domain works that allow them to evaluate AI output against a rich background of domain knowledge. The people who are least well positioned are the people who developed their skills in the presence of AI, who learned the domain through the mediation of automated systems rather than through direct engagement, who never built the pattern libraries that effective oversight requires.

This is the expertise paradox. The current generation of practitioners can oversee AI effectively because they built their expertise before AI arrived. The next generation of practitioners, trained in an AI-augmented environment, will lack the experiential foundation that makes effective oversight possible. The oversight capacity is a non-renewable resource — it was built under conditions that the technology is now eliminating, and there is no mechanism within the technology itself to regenerate it.

The paradox has a temporal dimension that makes it especially dangerous. The degradation of oversight capacity does not manifest immediately. In the first months and years of AI deployment, the current generation of experts provides effective oversight because their pattern libraries are still fresh, still being maintained through whatever residual direct engagement the new workflow allows. The AI-assisted workflow appears to work well — outputs are reviewed, errors are caught, the system performs as expected. Organizational leaders observe the smooth functioning and conclude that the system is reliable, that the oversight is adequate, that the workflow can be scaled.

The degradation manifests later, gradually, as the current generation of experts retires and is replaced by practitioners trained in the AI-augmented environment. The new practitioners are competent operators of the system. They are not, and cannot be, the kind of expert reviewers that the system's reliability depends upon. They lack the patterns. They lack the anomaly detection capability. They lack the mental models of the domain that would allow them to evaluate AI output against the deep understanding that only direct experience provides.

The degradation is invisible because the failures it produces are invisible. The errors that a Darlene would have caught are not caught by the next-generation nurse who never built Darlene's pattern library. But the errors that are not caught do not announce themselves as oversight failures. They present as bad luck, as statistical noise, as the irreducible uncertainty of a complex domain. The organizational metrics that monitor system performance may show no change, because the metrics were designed to capture the kinds of errors that the system itself detects, not the kinds of errors that only human expertise can detect.

This is what Klein's framework reveals about the AI transition that performance metrics cannot capture: the most dangerous failures are the ones that do not show up in the data. The infant whose sepsis was caught early by Darlene shows up in the data as a successful outcome. The infant whose sepsis was not caught because no one with Darlene's expertise was on the unit shows up in the data as a deterioration that began at a certain time — but the data cannot tell you that the deterioration could have been caught earlier if the oversight capacity had not degraded. The counterfactual — the error that would have been caught by an expert who no longer exists — is invisible to any measurement system.

Klein's career-long adversarial collaboration with Daniel Kahneman sharpens this analysis. Kahneman and Klein published their joint paper "Conditions for Intuitive Expertise: A Failure to Disagree" in 2009, establishing the conditions under which expert intuition can be trusted: the environment must provide valid cues that are reliably associated with outcomes, and the expert must have had sufficient opportunity to learn those cues through practice and feedback. When these conditions are met, Klein's research shows expert intuition to be remarkably reliable. When they are not met — when the environment is unpredictable, when feedback is absent or delayed, when the cues are misleading — expert intuition degrades into the biased, overconfident judgment that Kahneman's research documents.

The AI transition disrupts both conditions simultaneously. The first condition — valid cues — is preserved in principle but obscured in practice, because the AI system mediates the practitioner's relationship with the domain. The practitioner sees the AI output rather than the raw domain data. The cues she receives are filtered through the system's processing, which may preserve the informational content while stripping the contextual richness that human pattern recognition depends upon. The second condition — sufficient learning opportunity — is directly undermined by the automation of the tasks that provide the learning. If the AI handles the implementation and the practitioner handles the review, the practitioner receives feedback on the AI's performance but not the embodied, direct-engagement feedback that builds the pattern library.

The implications for organizational design are concrete and specific. Klein argues that organizations deploying AI must deliberately design experiences that build and maintain human pattern libraries, even when those experiences are less efficient than allowing the AI to handle the work. The military has a term for this: manual reversion training. Pilots who fly highly automated aircraft are required to practice manual flying at regular intervals, not because manual flying is more efficient but because the manual skills are needed when the automation fails. The requirement exists because the military learned, through catastrophic experience, that automation erodes the manual skills it depends upon for backup.

The analogy to AI-augmented knowledge work is direct. Organizations that deploy AI coding assistants need to ensure that their developers still write code by hand, regularly, in conditions that build and maintain the pattern libraries that effective code review requires. Organizations that deploy AI diagnostic tools need to ensure that their clinicians still examine patients directly, regularly, in conditions that build and maintain the perceptual skills that effective diagnostic oversight requires. Organizations that deploy AI legal research tools need to ensure that their lawyers still read cases closely, regularly, in conditions that build and maintain the legal reasoning skills that effective review of AI-generated briefs requires.

These requirements impose costs. Manual practice is slower than AI-assisted production. Time spent building pattern libraries is time not spent producing output. Organizations operating under quarterly earnings pressure, competitive dynamics that reward speed, and market incentives that penalize investment in capabilities whose returns are uncertain and long-term, will resist these requirements with the structural reliability that Klein's decades of organizational research would predict.

The expertise paradox, then, is not merely a cognitive challenge. It is an institutional one. The cognitive science is clear: human oversight of AI depends on human expertise, and human expertise depends on experiences that AI is eliminating. The institutional question is whether organizations will invest in preserving those experiences — accepting the short-term costs in exchange for long-term reliability — or whether they will follow the structural incentives of the market toward maximum efficiency and discover, too late, that the expertise upon which their systems depended has been consumed without replacement.

Klein's field research provides the diagnostic framework. The prescription follows from the diagnosis: design for expertise preservation the way engineers design for structural load. Not as an afterthought, not as a training program bolted onto an otherwise optimized workflow, but as a first-order design requirement that shapes the architecture of AI-augmented work from the beginning.

The fire commander's knowledge was built in burning buildings. The NICU nurse's knowledge was built at bedsides. The senior engineer's knowledge was built in codebases. Each of these environments is being transformed by AI in ways that may improve immediate outcomes while degrading the conditions under which the next generation of experts develops the judgment to maintain those outcomes. The paradox will not resolve itself through better technology. It will resolve through better organizational design — or it will not resolve at all, and the cost will be paid in failures that no one saw coming because no one was left who knew what to look for.

Chapter 5: The Smuggled Expertise Problem

In February 2024, Gary Klein published an essay with a title that functioned as a diagnostic instrument: "Spotting Exaggerated Claims for AI Superiority Over Experts." The essay was not a polemic. It was a field guide — a set of tools for practitioners who were being told, with increasing frequency and confidence, that the machines had surpassed them. Klein's tone was that of a researcher who had spent decades watching one set of claims about human cognition get replaced by another, and who had developed, through that experience, a practiced eye for the methodological sleights of hand that make the new claims appear more solid than they are.

The most important tool in Klein's field guide is the concept of smuggled expertise. The concept is simple to state and devastating in its implications. When a machine learning system is trained on data generated by human experts — medical records written by physicians, legal briefs drafted by lawyers, code written by engineers, diagnostic assessments produced by clinicians — the system's performance incorporates the expertise of the humans who generated the training data. The system does not replace that expertise. It consumes it. And when the system is then evaluated against human experts, the comparison is structurally unfair, because the system is being measured against the very people whose judgment it has already absorbed.

Klein illustrated the problem with a study that claimed a machine learning algorithm could predict emergency department patient outcomes more accurately than emergency physicians. The algorithm was trained on data from electronic health records — records that contained the observations, assessments, and clinical decisions of the ED physicians and nursing staff. The variables the algorithm used to make its predictions included vital signs, lab results, medication orders, and clinical notes, all of which reflected the judgment of the clinicians who had examined the patients, ordered the tests, and documented their findings. The algorithm was not reading the patients. It was reading what the clinicians had written about the patients. It was building its predictions on a foundation of clinical expertise that the study's design rendered invisible.

The smuggled expertise problem is not a technical flaw that better study design can eliminate. It is a structural feature of any system trained on expert-generated data. The training data is not raw observation. It is the product of human cognition — the selection of what to observe, the interpretation of what was observed, the judgment about what to record and how to record it. Every data point in a medical record reflects a clinical decision. Every line of code in a training corpus reflects an engineering judgment. Every legal citation in a brief reflects a lawyer's assessment of relevance. The expertise is embedded in the data at a level that cannot be extracted or controlled for, because the data would not exist without it.

This has implications that extend far beyond the methodological critique of individual studies. If the performance of AI systems depends, to a degree that is difficult to quantify but impossible to deny, on the expertise embedded in their training data, then the relationship between AI and human expertise is not competitive but parasitic in a precise technical sense. The AI system feeds on the expertise it appears to replace. Its performance is a function of the quality and richness of the human judgment that produced the data from which it learned.

The parasitic relationship creates a sustainability problem that Klein's framework identifies with particular clarity. If the deployment of AI systems reduces the number of human experts practicing in a domain, if it displaces the physicians whose clinical notes feed the algorithm, the engineers whose code trains the model, the lawyers whose briefs constitute the legal corpus, then the quality of the training data degrades over time. The expertise that the system consumed is not being regenerated, because the conditions under which it was generated — direct engagement with patients, hands-on coding, close reading of case law — are being eliminated by the system itself.

The degradation is not immediate. Training data accumulated over decades provides a deep reservoir of expertise that current systems can draw upon. But the reservoir is finite. If the inflow of new expert-generated data slows because fewer experts are practicing, or because the experts who remain are spending less time in direct engagement with the domain, the reservoir will eventually be drawn down. The AI system's performance will plateau, then degrade, as the training data becomes less representative of the evolving domain — new diseases, new legal precedents, new engineering challenges that were not present in the historical data.

Klein identifies a second structural problem in the AI-versus-expert comparison: the learning confound. In the studies he examined, the AI system was designed to learn from the data. It was given access to large datasets, sophisticated algorithms, and the computational resources to identify patterns that no human could detect through unaided cognition. The human experts, by contrast, were evaluated on their existing knowledge without being given comparable learning opportunities. They were not shown the data the algorithm had learned from. They were not given time to study the patterns the algorithm had identified. They were tested cold, on their clinical judgment as it stood at the moment of evaluation, against a system that had been optimized specifically for the task at hand.

The unfairness of this comparison is so stark that it should disqualify the conclusions drawn from it. Yet the studies are published, cited, and used to justify organizational decisions about the deployment of AI systems and the reduction of human expert involvement. The learning confound is invisible to readers who do not have Klein's trained eye for the methodological assumptions that shape research design, which is to say, it is invisible to the vast majority of decision-makers who encounter these studies in summary form.

The third problem Klein identifies — big-data intimidation — operates at the rhetorical rather than the methodological level, but it is no less consequential. The AI system in the emergency department study incorporated over sixteen thousand variables. The number is meant to impress, and it does. Sixteen thousand variables sounds like a comprehensiveness that no human clinician could match. But Klein's analysis reveals that the empirical optimum — the number of variables that actually contributed to predictive accuracy — was two hundred twenty-four. And performance plateaued at approximately twenty. The remaining fifteen thousand seven hundred eighty variables contributed noise rather than signal.

The gap between the number of variables the system used and the number it needed is not merely a technical detail. It reveals something about the rhetoric of AI performance claims. The impressive number is deployed not because it improves the system's predictions but because it improves the audience's confidence. It is a rhetorical device masquerading as a technical specification. The decision-maker who hears "sixteen thousand variables" receives a message about the system's superiority that the actual predictive architecture does not support.

Klein's field guide is, at its core, a toolkit for cognitive self-defense. It is designed for the practitioner who is being told that the machine is better than she is, who is being shown studies and metrics and performance comparisons that appear to support the claim, and who needs the analytical tools to evaluate whether the claim is warranted. The tools Klein provides — check for learning confounds, look for smuggled expertise, do not be cowed by big-data claims — are not technically sophisticated. They are diagnostically precise. They target the specific methodological vulnerabilities that make AI superiority claims appear more robust than they are.

The need for these tools is intensified by what Klein has described elsewhere as the asymmetry of confidence between AI systems and human experts. AI systems present their outputs with uniform confidence. A large language model does not hesitate, does not qualify, does not express the uncertainty that a human expert would convey through hedging language, facial expressions, or the characteristic pause that signals "I'm not sure about this." The output arrives fluent, complete, and structurally indistinguishable from the output produced when the system is operating within its domain of competence. The human reviewer, confronted with this confident output, must supply her own uncertainty — must ask herself whether the output is correct without receiving any signal from the system about whether it might not be.

This asymmetry exploits a cognitive vulnerability that Klein's research has documented extensively: the tendency to calibrate confidence to fluency. When information is presented fluently — clearly, confidently, without hesitation — the receiver tends to assign it higher credibility than information presented with hedges, qualifications, or visible uncertainty. In human interaction, the expert who says "I'm pretty sure, but let me check" signals both competence and epistemic humility. The AI system that presents its output without qualification signals only competence. The epistemic humility is absent, not because the system has chosen to suppress it but because the system does not possess it. It does not know what it does not know.

The practitioner who lacks Klein's diagnostic tools, who does not know to check for smuggled expertise or learning confounds, who is susceptible to big-data intimidation and the asymmetry of confidence, is the practitioner most likely to accept AI output without the critical evaluation that the output requires. She is also, increasingly, the practitioner who is being trained — the next-generation professional developing her skills in an environment where AI outputs are a constant presence, where the standard of performance is set by the machine, and where the evaluative skills that Klein's tools support are not explicitly taught because the curriculum has not yet adapted to the need.

The smuggled expertise problem is ultimately a problem of visibility. The expertise that AI systems consume is invisible in the training data because it has been naturalized — absorbed into the records, the code, the documents that constitute the data, indistinguishable from the raw information it was used to generate. The studies that compare AI to human experts render the consumed expertise invisible through methodological design — by failing to account for the human judgment embedded in the data the AI learned from. The organizational decisions that deploy AI systems render the ongoing need for human expertise invisible through performance metrics that capture the system's outputs without measuring the conditions under which those outputs can be trusted.

Making the invisible visible is the first step toward addressing the sustainability problem that smuggled expertise creates. Organizations that deploy AI systems need to ask, explicitly and repeatedly, where the expertise in the training data came from, whether the conditions under which it was generated are being maintained, and what happens when they are not. The question is not comfortable. It requires acknowledging that the impressive performance of the AI system is not self-generating — that it depends on a human infrastructure of expertise that the system's own deployment is eroding.

Klein's career has been built on making the invisible visible — on demonstrating that the rapid, confident decisions of experts are not mysterious intuitions but the products of cognitive processes that can be studied, understood, and, critically, supported or undermined by organizational design. The AI transition extends this project into new territory. The cognitive processes that Klein has studied are now operating in a new environment, one in which the most powerful tool the expert has ever encountered is simultaneously the greatest threat to the conditions under which her expertise was built.

The smuggled expertise problem will not be solved by better AI. It will be solved by organizations that understand what they are consuming and commit to regenerating it. The alternative is a system that performs well on yesterday's problems because it was trained on yesterday's expertise, and fails on tomorrow's problems because the expertise needed to address them was never developed.

Chapter 6: Mental Simulation and the Vanishing Rehearsal

The fire commander does not merely recognize the type of fire he is facing. He runs the response forward in his mind before committing his crew. He imagines ordering an interior attack, then watches the scenario unfold: the crew enters through the front, advances toward the seat of the fire, the hose line reaches the kitchen — but the floor in the hallway was soft, which means structural compromise underneath, which means the crew's weight could bring them through. The mental simulation catches the problem. The commander modifies the plan before a single firefighter crosses the threshold.

Klein documented this process across hundreds of critical incident interviews and identified it as the second phase of recognition-primed decision making. The first phase is pattern recognition — the rapid matching of current conditions to previously encountered situations. The second phase is mental simulation — the expert's capacity to project an action forward in time, imagining how the situation will evolve if the action is taken, watching for the moment when the projected scenario breaks down. When the simulation runs smoothly, the expert acts. When the simulation reveals a problem, the expert modifies the action or cycles to a different approach.

Mental simulation is not imagination in the colloquial sense. It is constrained projection — a process governed by the expert's model of how the domain works. The fire commander's simulation of the interior attack is constrained by his knowledge of structural engineering, fire behavior, hose dynamics, crew movement patterns, and the specific features of the building in front of him. The simulation is realistic because the underlying model is rich. A novice attempting the same simulation would produce a thinner, less accurate projection, because the novice's model of the domain is less developed. The quality of the simulation is a direct function of the depth of the pattern library.

Mental simulation serves a function in human decision-making that has no current analog in AI systems: it allows the expert to evaluate an action before taking it. The evaluation is not a calculation of probabilities. It is an experiential rehearsal — the expert lives through the scenario in compressed time, attending to the same cues she would attend to in real life, registering the same sensations of fit or misfit that would guide her in the actual situation. The simulation is, in an important sense, a form of practice. Each time the expert runs a simulation, she is exercising and refining the same cognitive capacities that direct engagement with the domain would exercise.

This function matters for the AI transition because the deployment of AI systems is systematically reducing the opportunities for the kind of experiential rehearsal that mental simulation provides. When the AI handles the implementation — writing the code, drafting the brief, generating the diagnosis — the practitioner does not run the implementation forward in her mind. She does not imagine the code executing, line by line, watching for the moment when the logic fails. She does not project the legal argument through the judge's likely responses, testing each move against the expected counter. She does not simulate the diagnostic pathway, imagining the patient's response to each test, watching for the result that does not fit. The AI produces the output, and the practitioner evaluates the output as a finished product rather than as a projected action whose consequences can be rehearsed.

The difference is not trivial. Evaluating a finished product is a different cognitive operation from simulating an action in progress. Evaluation is retrospective: the practitioner examines what was produced and assesses its quality. Simulation is prospective: the practitioner imagines what will happen and watches for problems. The prospective orientation engages a richer set of cognitive resources — the domain model, the pattern library, the anticipatory attention that detects misfit between expected and actual — because the practitioner is not merely assessing an artifact but inhabiting a scenario.

Consider the software engineer who, before AI, would write a function and then run it in her mind before executing it. She would trace the logic: if this input, then this variable takes this value, then this condition is met, then this branch executes — wait. That branch assumes the array is non-empty, but nothing upstream guarantees that. The mental simulation catches the edge case before the code runs. The catch is a product of the simulation, which is a product of the engineer's model of how the code will execute, which is a product of years of experience writing code and tracing its behavior.

With AI-assisted development, the engineer describes the function in natural language. Claude produces the code. The engineer reviews the code as a finished artifact. She may or may not trace the logic in her mind. If the code looks structurally sound and matches her expectation of what the function should do, she may accept it without performing the mental simulation that would have caught the edge case. The edge case is still there. The engineer's cognitive process no longer includes the step that would have found it.

The reduction in mental simulation is not something practitioners notice, because the simulation was never a conscious, deliberate step in their workflow. It was embedded in the act of writing — an automatic accompaniment to the process of converting intention into code, argument into brief, hypothesis into diagnostic plan. When the conversion is performed by the AI, the accompanying simulation simply does not occur. Its absence is invisible because it was never visible in the first place. The engineer does not think "I used to simulate and now I don't." She thinks "the code looks right." The absence of the simulation does not register as a loss. It registers as efficiency.

Klein's research on mental simulation reveals a further dimension of the problem. The capacity for mental simulation, like the pattern library on which it depends, is maintained through exercise. Experts who regularly engage in mental simulation — projecting actions, watching for breakdowns, modifying approaches based on simulated outcomes — maintain and sharpen the capacity. Experts who stop engaging in mental simulation — because the work that prompted it has been automated, because the pace of AI-assisted production does not leave time for it, because the organizational incentives reward output volume over deliberative depth — experience a gradual degradation of the capacity.

The degradation follows the same trajectory as the degradation of the pattern library, because the two are interdependent. Mental simulation depends on the pattern library for its content — the patterns provide the model of the domain that constrains and informs the projection. The pattern library depends on mental simulation for its refinement — each simulation that reveals a misfit between expected and actual contributes to the updating and correction of the stored patterns. The two cognitive processes form a cycle: patterns inform simulations, simulations reveal anomalies, anomalies update patterns. Interrupt either process, and the cycle degrades.

AI-assisted work interrupts both processes simultaneously. It reduces the direct experiences that build patterns, as documented in the preceding chapters. And it reduces the mental simulations that refine patterns, because the act of producing output — the act that naturally prompted mental simulation — has been transferred to the machine.

Klein described this dynamic in his 2021 analysis of criticisms and confusions surrounding the RPD model. He noted that it would be easy to build a computer simulation of RPD if one focused only on the pattern-matching component — training a system to match situations to stored patterns and generate associated actions. But the mental simulation component resists computerization, because the simulation depends on a rich, flexible, context-sensitive model of the domain that the expert has built through direct experience. The model is not a fixed data structure. It is a dynamic cognitive instrument that the expert manipulates in real time, adjusting parameters, testing assumptions, watching for consequences that her specific experiential history has taught her to anticipate.

Klein made a further observation that bears directly on the AI transition. He noted that people do not merely weaken connections when a simulation fails. They diagnose what went wrong in order to build stronger mental models. The diagnostic process — the effortful, conscious analysis of why the simulation broke down — is itself a mechanism of learning. AI systems that use reinforcement learning strengthen or weaken connections based on outcomes. Human experts construct explanations of failure that reshape their understanding of the domain. The explanatory process is deeper than the adjustment process. It produces not just a correction but an understanding — a revision of the mental model that makes future simulations more accurate across a wider range of situations.

This distinction between adjustment and understanding maps onto a distinction that runs through the entire AI transition debate. AI systems adjust. They modify their parameters in response to feedback, producing better outputs over time. Human experts understand. They construct models of why things work and why they fail, producing not just better outputs but a richer comprehension of the domain that supports generalization, transfer, and the creative application of knowledge to novel situations.

The creative application is where the loss of mental simulation matters most. Klein's research documents cases in which mental simulation served not just the evaluative function of checking an action for problems but the generative function of discovering new possibilities. A military commander running a simulation of a planned operation notices that a particular unit will arrive at a particular location at a particular time — and realizes that this coincidence creates an opportunity for a maneuver that was not part of the original plan. A surgeon simulating a procedure notices that the approach to the surgical site will expose a second area that could be addressed in the same operation, converting a single procedure into a more comprehensive intervention. An engineer simulating the execution of a system notices an interaction between components that suggests a simpler architecture than the one originally designed.

These discoveries are not the result of deliberate search. They are the byproducts of mental simulation — the unexpected consequences of running a rich model forward in time and attending to what emerges. They depend on the same cognitive infrastructure that supports the evaluative function: the pattern library, the domain model, the anticipatory attention that detects significant features of the projected scenario. Remove the simulation, and the discoveries do not occur. Not because the opportunities are not there, but because the cognitive process that would have noticed them is no longer being engaged.

The AI transition promises to make work faster, more productive, more efficient. These promises are substantially fulfilled. The work is faster. The output is greater. The efficiency, measured in units of output per unit of input, is dramatically improved. What the efficiency metrics do not capture is the cognitive activity that the efficiency displaces — the mental simulations that are not run, the anomalies that are not detected, the discoveries that are not made, the understanding that is not built.

Klein's framework suggests that the cost of this displacement will not be visible in the short term, because the current generation of practitioners still carries the cognitive infrastructure built through pre-AI experience. The cost will manifest later, gradually, as the infrastructure degrades through disuse and is not rebuilt in the next generation. The manifestation will take the form not of dramatic failures but of a slow erosion of the creative and evaluative capacities that distinguish expert performance from competent operation — a flattening of the cognitive landscape that produces adequate work without the depth of understanding that turns adequate into excellent, or that catches the subtle error before it becomes catastrophic.

The vanishing rehearsal is not a metaphor. It is a description of a cognitive process — specific, documented, empirically grounded — that is being removed from the workflow of expert practitioners by the efficiency of AI-assisted production. The removal is not deliberate. It is structural. And the consequences, like the consequences of all structural changes to the conditions of expert development, will be borne by the people who were never given the opportunity to build what they will someday need.

Chapter 7: The Trust Calibration Problem

In the early 1990s, an Air Force colonel in the Pentagon delivered a pronouncement that Gary Klein has cited for three decades as an emblem of institutional failure. Frustrated by widespread resistance to AI systems designed to assist military professionals, the colonel declared: "They'll just have to learn to trust the system." The pronouncement was directed at pilots, maintenance technicians, and logistics specialists who had been pushing back against the AI tools foisted on them — tools whose recommendations they could not understand, whose reasoning they could not follow, and whose errors they could not detect.

The colonel's pronouncement treated trust as a compliance problem. The users did not trust the system. The solution was to make them trust it — through training, through mandates, through the institutional authority that a colonel's stars conferred. The possibility that the users' distrust was informative, that it reflected a cognitively sophisticated assessment of the system's limitations, did not enter the colonel's framing. The users were obstacles to deployment. Their resistance was a problem to be overcome rather than a signal to be understood.

Klein recognized in that pronouncement a pattern he would encounter repeatedly over the following decades: the institutional tendency to treat the human relationship with automated systems as a trust problem rather than a calibration problem. Trust, in the institutional framing, is binary — the user either trusts the system or does not. The solution to insufficient trust is more trust. The solution to excessive trust is never articulated, because institutions that have invested in AI systems are structurally incapable of advocating for less trust in those systems.

Klein's alternative framing — trust calibration — is more demanding and more useful. The goal is not more trust or less trust but appropriate trust: trust that is calibrated to the system's actual performance in the specific situation at hand. Appropriate trust means trusting the system when it is operating within the boundaries of its demonstrated competence and distrusting it when it is not. Calibrated trust requires the user to have a mental model of the system's competence — an understanding of the conditions under which it performs well, the conditions under which it fails, and the boundary between the two.

The requirement for a mental model of the system's competence is where Klein's analysis becomes most consequential for the AI transition. Building a mental model of any complex system requires experience with the system's behavior across a range of conditions, including, critically, conditions under which the system fails. The user who has seen the system fail — who has observed the specific failure modes, the patterns of error, the situations that produce unreliable output — has the experiential foundation for calibrated trust. She knows what the system is good at and what it is not. She can adjust her reliance accordingly, trusting the output in familiar conditions and scrutinizing it in unfamiliar ones.

The user who has not seen the system fail, who has interacted with the system only under conditions where it performs well, has no basis for calibration. Her trust is either wholesale — she trusts because she has no reason not to — or it is anchored to the wrong features, calibrated to the fluency of the output rather than the accuracy of the content, to the confidence of the presentation rather than the reliability of the reasoning.

The DARPA XAI program, which Klein's team supported, was an institutional recognition that the trust calibration problem was real and consequential. The program's premise was that AI systems deployed in military contexts needed to be explainable — not merely accurate but capable of providing explanations that would support the users' construction of mental models. If the user could understand how the system arrived at its recommendation, she could evaluate whether the reasoning was sound in the current situation, which would support calibrated trust.

Klein's team discovered that explanation, as typically conceived in the AI research community, was insufficient for calibration. The AI teams in the XAI program focused on producing local explanations — accounts of why the system made a particular prediction in a particular case, typically by highlighting the features that most influenced the output. Klein's team found that local explanations, while useful, did not support the construction of the global mental model that calibrated trust requires. Knowing why the system made this prediction does not tell the user when the system is likely to make wrong predictions. The user needs not just a window into the system's reasoning on individual cases but a map of the system's competence boundary — an understanding of where the boundary lies and what features of the situation indicate whether she is inside or outside it.

The AIQ toolkit was designed to support this boundary-mapping function. Rather than explaining individual outputs, it helped users identify the conditions under which the system was likely to perform well and the conditions under which it was likely to fail. The toolkit included exercises that exposed users to the system's failure modes — deliberate presentations of cases where the system produced incorrect or misleading outputs — so that users could build the experiential foundation for recognizing similar conditions in the future.

The design philosophy behind AIQ reflects Klein's foundational insight: expertise is built through exposure to failure, not just success. The fire commander who has only seen fires that behaved as expected has not built the anomaly detection capacity that allows him to recognize when a fire is behaving unexpectedly. The pilot who has only flown in calm conditions has not built the manual flying skills that allow her to recover when automation fails. The AI user who has only seen the system produce correct outputs has not built the calibration capacity that allows her to recognize when the system is producing plausible nonsense.

This insight has immediate practical implications for how organizations deploy AI systems. The standard deployment model emphasizes the system's strengths: demonstrations of impressive performance, case studies of successful application, metrics showing improvement over human baselines. The standard deployment model builds uncalibrated trust, because it exposes users to the system's success without exposing them to its failure. The result is a population of users who trust the system too much in conditions where trust is not warranted and who lack the experiential foundation for detecting the system's errors.

Klein's alternative would be a deployment model that deliberately includes exposure to the system's failure modes. Before relying on the system in production, users would encounter curated examples of the system's errors — cases where it produced outputs that were plausible, confident, and wrong. Users would practice detecting these errors, developing the pattern recognition capacity for identifying the cues that signal the system is operating outside its competence boundary. The practice would build the experiential foundation for calibrated trust, turning users from passive recipients of system output into active evaluators whose trust is modulated by the situation.

The alternative deployment model is rare in practice, for reasons that Klein's organizational research would predict. It is more expensive than the standard model. It requires more time. It exposes the system's weaknesses rather than showcasing its strengths, which creates discomfort for the vendors and internal champions who are advocating for deployment. And it produces a user population that is more skeptical of the system — which, from the perspective of calibrated trust, is the goal, but from the perspective of adoption metrics, looks like failure.

The organizational incentives, in other words, are structurally opposed to the conditions that calibrated trust requires. The market rewards rapid adoption. Calibrated trust requires slow exposure. The market rewards confident users. Calibrated trust requires appropriately skeptical users. The market rewards seamless integration. Calibrated trust requires deliberate friction — the friction of encountering the system's failures, of building the mental model of its competence boundaries, of developing the evaluative skills that make human oversight substantive rather than ceremonial.

The trust calibration problem intersects with the expertise paradox described in the preceding chapters. The practitioners who are best calibrated — who have the richest mental models of their AI systems' competence boundaries — are the practitioners who brought domain expertise to the interaction. They can map the system's competence boundary because they know the domain well enough to recognize when the system's output departs from domain reality. The practitioners who are least calibrated are the practitioners who lack domain expertise, who cannot evaluate the system's output against a rich understanding of the domain, and who therefore cannot distinguish between the conditions where the system is reliable and the conditions where it is not.

As the expertise paradox predicts, the proportion of well-calibrated users will decline over time if the conditions under which domain expertise is built are being eroded by the AI deployment itself. The current generation of users, trained before AI, brought calibration capacity to the interaction. The next generation, trained with AI, will bring less. The generation after that, trained entirely within AI-mediated environments, may bring almost none. Each generation's calibration capacity will be thinner than the last, because each generation will have had fewer direct experiences with the domain from which calibration is built.

The trajectory is a slow-motion institutional failure of the kind that Klein has documented across dozens of high-stakes domains. The failure does not manifest as a single catastrophic event. It manifests as a gradual increase in the frequency of undetected errors — errors that a well-calibrated user would have caught, that a poorly calibrated user accepts because she lacks the basis for skepticism, that accumulate in the system's outputs without triggering the alarms that would have been triggered if the alarms — the human experts — had been properly maintained.

Klein's work points toward a principle that organizations deploying AI would do well to adopt: trust calibration is not an initial training exercise that can be completed and checked off. It is an ongoing maintenance requirement, analogous to the ongoing maintenance of the beaver's dam that Edo Segal describes in The Orange Pill. The calibration must be refreshed through regular exposure to the system's failure modes, through deliberate practice in error detection, through organizational structures that reward skepticism rather than penalizing it. The moment the maintenance stops, the calibration begins to degrade, and the degradation follows the same invisible, gradual, cumulative trajectory as the degradation of the pattern library and the atrophy of mental simulation.

The Air Force colonel who demanded that users "learn to trust the system" was proposing the opposite of what Klein's research prescribes. He was proposing uncalibrated trust — trust without understanding, compliance without evaluation, adoption without the cognitive infrastructure that makes adoption safe. Three decades later, the proposition is being made again, at a scale the colonel could not have imagined, by organizations deploying AI systems across every domain of professional practice.

The proposition is still wrong. And the cost of its wrongness is still being borne by the practitioners who are told to trust systems they cannot evaluate, in conditions they cannot assess, on the basis of demonstrations that showed them only the system's strengths and never its failures.

Chapter 8: The Pre-Mortem and the Social Architecture of Foresight

In April 2025, Gary Klein published an essay asking a question that went to the heart of his life's work: Can AI do pre-mortems for us? The pre-mortem is Klein's most widely adopted practical invention — a structured technique in which a team imagines that a project has already failed and works backward to identify the causes. The technique, developed in the 1990s, has been adopted across military planning, corporate strategy, medical safety, and, in recent years, AI risk assessment. Its appeal lies in its elegant inversion of normal planning psychology: instead of asking "how will we succeed?" the pre-mortem asks "we have failed — why?"

The inversion works because it exploits a known feature of human cognition. When people evaluate a plan they have developed, they are subject to confirmation bias — the tendency to seek and interpret information in ways that confirm the plan's viability. The plan is theirs. They are invested in it. The cognitive effort of developing the plan has created an attachment that makes disconfirming information psychologically costly to process. The pre-mortem circumvents this bias by changing the frame. The project has already failed. The failure is a given. The team's task is not to defend the plan but to explain the failure. The framing gives permission to identify problems that the normal planning process would suppress.

Klein acknowledged in the essay that large language models produce pre-mortem outputs that are, by his own assessment, "surprisingly good." Feed a project plan to an LLM, ask it to imagine the project has failed, and request a list of potential causes. The output will be coherent, comprehensive, and plausible. It will identify risks that a human team might overlook, drawing on a broader base of analogous failures than any single team's experience could provide. The output will be generated in minutes rather than the hour or more that an in-person pre-mortem requires. By the metrics that organizations typically use to evaluate process efficiency — speed, breadth, cost — the AI pre-mortem is superior.

Klein's essay then did what Klein's writing always does: it looked past the metrics to the cognitive and social reality they obscure. The in-person pre-mortem, Klein observed, is not merely a technique for generating risk lists. It is a social process that serves functions the risk list does not capture.

The first function is psychological safety. The pre-mortem's hypothetical framing — "imagine the project has failed" — creates a protected space in which team members can voice concerns they would not raise in a normal planning meeting. A junior team member who has noticed a flaw in the plan proposed by a senior colleague faces a social cost for pointing out the flaw directly. The pre-mortem's frame removes the cost. She is not criticizing the plan. She is explaining why the project failed. The criticism is embedded in a hypothetical that distributes responsibility across the entire team rather than locating it in a confrontation between two individuals.

The AI pre-mortem eliminates this function entirely. The LLM does not feel social pressure. It does not hesitate to name risks. It does not need the protective framing that makes the pre-mortem psychologically safe for humans. But the elimination of the social function does not mean the social function was not needed. The team members who would have found, in the pre-mortem, the safety to voice their concerns now have no structured occasion for voicing them. The concerns do not disappear. They are suppressed — pushed back below the threshold of expression because the organizational context no longer provides the frame that would have brought them to the surface.

The second function is team calibration. Klein observed that experienced leaders use the pre-mortem not just to identify risks but to assess their teams. Which team members identify which risks reveals the team's distribution of expertise, attention, and concern. A leader who notices that no one on the team identified a particular category of risk — say, regulatory compliance, or supply chain vulnerability — learns something about the team's blind spots that the risk list itself does not reveal. A leader who notices that a particular team member consistently identifies risks that others miss learns something about that team member's value that performance metrics might not capture.

The AI pre-mortem produces the risk list without the team calibration. The leader receives the output without the observational opportunity. She knows what risks the AI identified. She does not know what risks her team would have identified, which team members would have identified them, or where the team's attention is concentrated and where it is absent. The informational content — the list of risks — is preserved. The relational content — the understanding of the team — is lost.

Klein cited a colleague who identified a third function: the pre-mortem as a trust-building exercise. Going through the pre-mortem together, sharing concerns, hearing each other's perspectives on what might go wrong, creates a shared experience of vulnerability and mutual acknowledgment that strengthens the team's capacity for collaboration under pressure. When the project encounters real difficulties, the team that has pre-mortemed together has a shared framework for interpreting the difficulty, a shared vocabulary for discussing it, and a shared experience of having anticipated it. The team that received an AI-generated risk list has none of these shared cognitive resources.

Klein's conclusion was precise and far-reaching: "One lesson here is how AI can devolve social tasks and coordination into data tabulations — and we may be poorer for it." The verb "devolve" is carefully chosen. It does not mean degrade or diminish. It means reduce to a lower level of organization — transform a complex social process into a simpler informational one. The devolution preserves the informational output while stripping away the social, relational, and organizational processes that made the output valuable in context.

The pre-mortem case is a microcosm of a much larger phenomenon that Klein's framework illuminates across the AI transition. In domain after domain, AI systems are capable of producing the informational output of complex human processes while being structurally incapable of reproducing the social processes that gave the output its meaning and value. The AI can generate the risk list but not the team calibration. It can produce the diagnosis but not the clinical relationship that contextualizes it. It can draft the legal brief but not the professional judgment about which arguments the judge will find persuasive given the specific history of the case and the specific inclinations of the bench. It can write the code but not the architectural understanding that determines whether the code belongs in the system at all.

In each case, the informational dimension of the work is separable from the social dimension. The AI handles the informational dimension with impressive speed and breadth. The social dimension — the trust-building, the team calibration, the relationship-based judgment, the contextual understanding that comes from being embedded in a specific organizational and interpersonal reality — falls away. The loss is invisible in the metrics because the metrics measure the informational output. The social processes that were bundled with the informational output were never measured, because they were never visible as separate processes. They were embedded in the work itself — byproducts of doing the work together, face to face, over time.

Klein's research on sensemaking, discussed in a preceding chapter, established that effective interpretation of ambiguous situations depends on both cognitive and social processes. The pre-mortem analysis extends this finding to the domain of foresight — the capacity to anticipate problems before they occur. Foresight, like sensemaking, has an irreducibly social dimension. The team that engages in collective foresight — sharing perspectives, identifying blind spots, building shared mental models of what might go wrong — develops a capacity for anticipatory coordination that no individual, and no AI system, can replicate.

The loss of this capacity has consequences that extend beyond the specific output of the pre-mortem. Teams that regularly engage in collective foresight develop what organizational researchers call shared mental models — aligned understandings of the task, the environment, and each other's roles and capabilities that enable coordination under pressure. When the unexpected occurs — when the plan encounters a reality that the plan did not anticipate — the team with shared mental models can adapt fluidly, because each member has an accurate model of what the other members know, expect, and are likely to do. The team without shared mental models must coordinate explicitly, through communication that consumes time and cognitive bandwidth that the situation may not allow.

The AI pre-mortem does not build shared mental models. It produces a document that team members may read individually, without the shared experience of constructing the analysis together. The document may be excellent. It may identify risks that the team would have missed. But it does not create the cognitive alignment that collective foresight produces — the shared understanding that enables the team to respond effectively when the risks materialize.

Klein's framework suggests that the organizations most likely to succeed in the AI transition are the ones that recognize the distinction between informational output and social process, and that preserve the social processes even when the informational output can be generated more efficiently by AI. These organizations will use AI to augment the pre-mortem — to provide a broader base of analogous failures, to identify risks that the team's experience might not cover, to challenge the team's assumptions with data the team has not considered — while preserving the in-person process through which the social functions are performed.

The distinction maps onto a broader principle in organizational design: the recognition that many of the most valuable functions performed by human teams are bundled with activities that appear, from the outside, to be purely informational. The meeting that produces a decision also produces alignment. The review that evaluates a product also builds shared standards. The mentoring session that transfers knowledge also builds trust. When AI handles the informational dimension of these activities — generating the decision options, evaluating the product, transferring the knowledge — the social dimensions that were bundled with them must be preserved through deliberate design rather than left to atrophy through neglect.

This is organizational design at its most demanding, because it requires leaders to value processes whose outputs cannot be measured and to invest in activities whose returns are indirect and long-term. The market rewards efficiency, and efficiency, in the AI era, means maximizing the informational output per unit of time while minimizing the human interaction that slows the process down. Klein's research suggests that the human interaction that slows the process down is, in many cases, the process that makes the informational output useful — the social architecture of foresight, sensemaking, and trust calibration without which the output is data rather than intelligence.

The pre-mortem was designed to exploit a feature of human cognition — the tendency toward confirmation bias — by creating a social structure that worked around it. The AI pre-mortem exploits the same feature of human cognition — the tendency to accept plausible outputs without scrutiny — without providing the social structure that would counteract it. The team that receives an AI-generated risk list is, in some ways, more vulnerable to confirmation bias than the team that never received one, because the list creates the illusion of comprehensive foresight without the reality of collective engagement that comprehensive foresight requires.

Klein's career-long insistence on studying cognition in its natural context — in the fire station, in the ICU, in the command post, rather than in the laboratory — has always carried an implicit argument about the social embeddedness of intelligent performance. The expert does not think alone. She thinks within a network of colleagues, mentors, institutional structures, and organizational cultures that shape what she attends to, what she considers, and what she decides. Remove the network, and the thinking changes — not because the individual's cognitive capacity has diminished but because the social infrastructure that supported, challenged, and calibrated her cognition is no longer there.

The AI transition is restructuring this social infrastructure at a speed that Klein's research has never before confronted. The restructuring is not deliberate. It is the structural consequence of optimizing for informational output in a technological environment that makes informational output extraordinarily cheap. The social processes that were bundled with the informational work — the pre-mortems, the mentoring conversations, the team debriefs, the casual interactions that build shared mental models — are collateral losses of an optimization that does not recognize their value because it does not measure their contribution.

Klein's pre-mortem essay ends not with a prescription but with an observation. The observation is that we may be poorer for the devolution of social processes into data tabulations. The conditional — "may be" — is characteristic of Klein's epistemic modesty. But the decades of research behind the observation leave little room for doubt about the direction. The social architecture of foresight is not a luxury. It is a load-bearing structure. And the organizations that strip it away in the name of efficiency will discover, when the structure is needed most, that it is no longer there.

Chapter 9: Cognitive Flexibility and the Architecture of Insight

Klein's research took a turn in the early 2000s that his colleagues in the naturalistic decision-making community did not entirely anticipate. He had spent two decades studying how experts make rapid decisions by recognizing patterns. Now he wanted to study something that pattern recognition alone could not explain: how experts see things that others miss. Not faster recognition of familiar situations, but the detection of something genuinely new — an opportunity, a connection, an anomaly that reshapes the expert's understanding of the situation in a way that opens possibilities that were not visible before.

The result was a research program on insight that produced, among other contributions, the book Seeing What Others Don't and a taxonomy of how insights arise in natural settings. Klein's method was characteristically empirical: he collected over a hundred and twenty cases of insight from published accounts, interviews, and historical records, and analyzed them for common cognitive structures. The insights ranged from scientific breakthroughs to firefighting decisions to police investigations to medical diagnoses. What Klein found was that insights are not random bolts of inspiration. They follow identifiable cognitive paths, and the paths depend on the same experiential foundation that supports pattern recognition and mental simulation — but they deploy that foundation in a different mode.

Klein identified three primary paths to insight. The first is connection — the detection of a link between two pieces of information that had not previously been associated. The second is contradiction — the recognition that something in the current situation does not fit the expected pattern, prompting a reinterpretation that reveals a new understanding. The third is creative desperation — the abandonment of a failing approach under extreme pressure, producing a radical reframing of the problem that opens a solution path invisible from within the original frame.

Each path depends on the expert's pattern library, but in a way that distinguishes insight from routine recognition. In routine recognition, the pattern library provides a match: the current situation activates a stored pattern, and the associated action follows. In insight, the pattern library provides the foundation for detecting a departure from expectation — a connection that should not exist, a contradiction that does not fit, a frame that is failing — and the departure triggers a cognitive process that reconfigures understanding. The pattern library is necessary for insight, but insight requires something more: the cognitive flexibility to abandon or restructure the patterns when the evidence demands it.

Klein calls this cognitive flexibility, and he treats it as a distinct capacity from pattern recognition, though built on the same experiential foundation. The expert who recognizes a familiar situation is deploying her patterns as templates — matching current conditions to stored cases. The expert who achieves insight is deploying her patterns as points of departure — using the expected pattern as a reference against which the unexpected can be detected, and then allowing the unexpected to restructure her understanding. The first operation is convergent: it narrows toward a recognized category. The second is divergent: it opens toward a new interpretation.

The distinction has direct implications for the AI transition, because the two operations are differently affected by the automation of domain work. Pattern recognition is the cognitive function that AI most closely approximates. Large language models are, at their core, pattern-matching systems operating at enormous scale. They excel at convergent operations — generating outputs consistent with the statistical regularities of their training data. When the situation falls within the distribution of the training data, the model's output is often indistinguishable from expert performance.

Cognitive flexibility — the capacity to detect when the patterns are failing and to restructure understanding in response — is the cognitive function that AI most conspicuously lacks. When a large language model encounters a situation that falls outside the distribution of its training data, it does not recognize the departure. It continues to generate pattern-consistent output, output that may be fluent and confident and wrong in precisely the way that Klein's research predicts: wrong because the situation requires a new frame, and the system can only apply existing ones.

The implications for the human practitioners who work alongside AI are troubling. If AI handles the routine pattern-matching work — the cases where the situation fits the expected template and the appropriate response is to apply the recognized pattern — then the practitioner's role shifts toward the non-routine cases, the cases that require cognitive flexibility, insight, the detection of departures from expectation and the construction of new interpretations. This is the shift that The Orange Pill describes when it argues that AI elevates human work to higher cognitive floors.

But the shift creates a developmental problem that Klein's insight research illuminates with particular clarity. Cognitive flexibility is not a general capacity that can be developed in the abstract. It is built through exposure to the specific domain in which flexibility is required. The expert who achieves insight in firefighting does so because her pattern library is rich enough to generate specific expectations against which specific departures can be detected. The richness of the expectation determines the precision of the departure detection. A thin pattern library generates vague expectations, which means departures must be dramatic to be noticed. A rich pattern library generates precise expectations, which means even subtle departures can be detected.

If the routine work that builds the pattern library is automated, the next generation of practitioners will arrive at the non-routine cases with pattern libraries that are too thin to support the cognitive flexibility the cases require. They will be asked to exercise insight in a domain they have not deeply experienced. They will be expected to detect departures from patterns they never fully learned. They will be required to restructure understanding they never fully built.

This is not a problem that can be solved by training practitioners to "think flexibly" in the abstract. Klein's research is unambiguous on this point: cognitive flexibility in a specific domain depends on deep experience in that domain. The firefighter does not achieve insight about fire behavior by studying creativity in a classroom. She achieves it by attending thousands of fires, building a pattern library so rich that the slightest deviation from expectation triggers a cognitive alarm. The alarm is the beginning of insight. Without the alarm, the deviation goes unnoticed, and the insight never occurs.

Klein's insight research also reveals a feature of expert cognition that has particular relevance to the AI transition: the role of what he calls the prepared mind. Pasteur's famous dictum — "chance favors the prepared mind" — is, in Klein's framework, not a platitude but a description of the cognitive preconditions for insight. The prepared mind is one that has accumulated enough domain experience to generate rich, precise expectations, and that has developed the cognitive flexibility to recognize when those expectations are violated and to use the violation as a starting point for new understanding.

The AI transition is creating conditions under which fewer minds will be prepared in this sense. The preparation requires the same direct, friction-rich engagement with the domain that builds the pattern library and maintains the capacity for mental simulation. If the engagement is reduced — if the routine work through which preparation occurs is automated — then fewer practitioners will arrive at the moments that demand insight with the cognitive preparation that insight requires.

Klein has documented cases in which insight failure produced catastrophic consequences. The Mann Gulch fire disaster, which he analyzed in detail, was a failure not of pattern recognition but of cognitive flexibility. The smokejumpers recognized the fire. They had patterns for routine fires. What they failed to do was recognize that the situation required a new frame — that the fire had shifted from routine to lethal — until the evidence was overwhelming and the time for response had nearly expired. Wagner Dodge, the crew's foreman, achieved the insight that saved his life: he lit an escape fire, creating a burned area in which he could lie down while the main fire passed over him. None of his crew members followed him, because they lacked either the cognitive flexibility to understand what he was doing or the trust to follow an action that contradicted every pattern they possessed. Thirteen of the sixteen smokejumpers died.

Dodge's escape fire is Klein's paradigm case of creative desperation — insight under extreme pressure, produced by the abandonment of a failing frame and the improvisation of a radically new approach. The insight depended on Dodge's deep experience with fire behavior, which gave him the understanding that a burned area would not re-burn — a fact that his less experienced crew members may not have possessed at a visceral, intuitive level. The experience was the foundation. The flexibility was the capacity to use the foundation in a way that no training manual had prescribed.

The AI transition is not creating Mann Gulch scenarios in the literal sense. But it is creating conditions under which the cognitive capacities that separate survivors from casualties in such scenarios — the deep experience, the rich expectations, the flexibility to abandon failing frames and improvise new ones — are systematically being eroded in the population of practitioners who will eventually face situations that demand them.

Klein's taxonomy of insight suggests that each of the three paths — connection, contradiction, creative desperation — is vulnerable to the atrophy of the experiential foundation on which it depends. Connections require a rich associative network built through diverse domain experience; if the experience is narrowed by AI-mediated work, the network thins. Contradictions require precise expectations against which deviations can be detected; if the expectations are never fully developed because the pattern-building work was automated, the deviations go unnoticed. Creative desperation requires the deep domain knowledge that supports improvisation under pressure; if the knowledge was never fully acquired, the improvisation has nothing to draw upon.

The organizations that will navigate the AI transition most effectively are the ones that design for insight as a first-order requirement — not just pattern recognition, not just efficient output, but the cognitive flexibility that allows practitioners to see what others miss, to detect the anomaly that does not fit, to restructure understanding when the situation demands it. Designing for insight means preserving the conditions under which rich pattern libraries are built, under which mental simulation is regularly practiced, under which practitioners encounter enough of the domain's variety to develop the precise expectations that make departure detection possible.

The fire at Mann Gulch burned itself out in minutes. The institutional conditions that determine whether the next generation of practitioners will be prepared for the moments that demand insight are being established now, through decisions about how AI is deployed, how training is structured, how work is organized. These decisions will determine not just how efficiently the routine work is handled but whether the non-routine — the situation that has never been seen before, that demands flexibility, insight, the capacity to see what others miss — will find, when it arrives, practitioners capable of meeting it.

Klein's career has been a sustained argument that human expertise is not a historical artifact to be superseded by computation but a cognitive achievement to be understood, valued, and preserved. The argument has never been more urgent than it is now, when the technology that most depends on expert oversight is simultaneously the technology that most threatens the conditions under which expertise develops. The insight that saves the infant, the building, the mission, the company — that insight depends on a foundation of experience that must be deliberately maintained against the current of efficiency that would sweep it away.

---

Chapter 10: Designing for Expertise in the Age of AI

Klein spent four decades documenting how expertise works. The final question his framework poses for the AI transition is not diagnostic but prescriptive: given everything that is known about how expertise develops, how it operates, and how it degrades, what must organizations do to preserve the conditions under which human expertise can flourish alongside AI systems that simultaneously depend on it and undermine it?

The question is not academic. It is the most practical question facing every organization that deploys AI tools, from the twenty-person startup to the multinational enterprise to the military command structure. And the answers, drawn from Klein's research across dozens of high-stakes domains, converge on a set of design principles that are consistent, specific, and uncomfortable for organizations that have optimized for efficiency above all else.

The first principle is deliberate exposure to the raw domain. Klein's research demonstrates that expertise is built through direct engagement with the domain's phenomena — the fires, the patients, the codebases, the legal cases, the operational environments where the domain's regularities and irregularities present themselves in their full, unmediated complexity. The mediation of AI filters the domain's phenomena through a processing layer that may preserve informational content while stripping the contextual richness on which pattern recognition depends. An AI diagnostic summary contains the relevant data points. It does not contain the visual gestalt of the patient's appearance, the nuance of the patient's description of symptoms, the temporal sequence of the patient's deterioration as observed over hours by the attending clinician.

Designing for deliberate exposure means building into AI-augmented workflows regular periods where practitioners engage with the domain without AI mediation. The software engineer writes code by hand, regularly, confronting the errors and edge cases that the AI would have handled invisibly. The clinician examines patients directly, regularly, building the perceptual skills that AI summaries cannot transmit. The lawyer reads cases closely, regularly, developing the interpretive capacity that AI research tools shortcut. These periods are not efficient. They are investments in the cognitive capital on which the efficiency of AI-augmented work ultimately depends.

The military analogy that Klein invoked in his DARPA work is instructive here. Fighter pilots who fly highly automated aircraft are required to practice manual flying at specified intervals. The requirement exists because the automation degrades the manual skills it was designed to augment. Pilots who rely exclusively on automated flight lose the capacity for the manual interventions that save lives when automation fails. The manual practice requirement is costly — it consumes flight hours, fuel, and training time that could be spent on other priorities. But the cost of not requiring it was demonstrated, catastrophically, in accidents where pilots who had lost manual skills could not recover from automation failures.

The second principle is structured failure exposure. Klein's trust calibration research demonstrates that appropriate trust in AI systems depends on the user's experience with the system's failure modes. The AIQ toolkit was designed to provide this experience in a controlled setting — presenting users with curated examples of system failures so that they could build the mental models needed to detect similar failures in practice. The principle extends beyond individual trust calibration to organizational learning: teams that regularly encounter AI failures in low-stakes, structured settings develop collective competence in error detection that teams insulated from failure do not.

Structured failure exposure means deliberately creating situations in which AI systems produce incorrect or misleading outputs and asking practitioners to detect the errors. These exercises can be built into regular training — periodic assessments in which the AI's output is intentionally wrong and the practitioner's task is to identify the error and explain why it is wrong. The exercises build the pattern library for AI failure modes, which is a different pattern library from the one built through domain experience but equally important for the practitioner's role as an oversight agent.

The design of failure exposure must be thoughtful. Random errors are less instructive than systematic ones. The most valuable exercises present the kinds of errors the AI system is actually prone to — the plausible hallucinations, the subtle misapplications, the confident assertions that are almost right but wrong in ways that matter. The exercises should mimic the conditions under which errors actually occur: time pressure, information overload, the seductive fluency of well-structured output. The goal is not to make practitioners paranoid about AI but to develop the calibrated skepticism that Klein's research identifies as the hallmark of effective human-AI collaboration.

The third principle is the preservation of social cognitive infrastructure. Klein's pre-mortem analysis demonstrated that AI can replicate the informational output of collective cognitive processes while eliminating the social processes through which teams build shared understanding, calibrate trust, and develop the relational knowledge that enables coordination under pressure. The principle applies broadly: mentoring relationships, team debriefs, case conferences, design reviews, and the informal interactions through which practitioners learn from each other's experience are social cognitive processes whose value is not captured by the informational outputs they produce.

Designing for social cognitive infrastructure means protecting these processes against the pressure to optimize them away. It means conducting in-person pre-mortems even though the AI version is faster. It means maintaining mentoring programs even though the AI can answer the mentee's questions more quickly. It means holding design reviews that bring the team together physically, around the same table, where the social dynamics of agreement and disagreement and persuasion and doubt can operate in the way that Klein's decades of field research have shown they must.

The protection will be costly. Every hour spent in an in-person pre-mortem is an hour not spent on AI-augmented production. Every mentoring session is a period of reduced throughput. The costs are visible and immediate. The benefits are invisible and long-term. And organizations, as Klein's research repeatedly demonstrates, are structurally biased toward visible, immediate costs and against invisible, long-term benefits.

The fourth principle is what Klein might call expertise auditing — the regular assessment of whether the organization's human expertise is being maintained, developed, or degraded by its AI deployment. The assessment would track not just the performance metrics of AI-augmented work but the cognitive health of the practitioners who perform it. Can the senior engineers still debug code manually? Can the clinicians still perform physical examinations with diagnostic confidence? Can the lawyers still construct legal arguments from first principles rather than from AI-generated templates?

These are not comfortable questions for organizations to ask, because the answers may reveal that the efficiency gains of AI adoption have come at the expense of the human capabilities on which the organization's long-term reliability depends. But the questions must be asked, regularly and honestly, because the degradation of expertise is invisible until it is catastrophic. Klein's research across dozens of domains demonstrates that the decline from expert to competent, from competent to adequate, from adequate to dangerous, follows a trajectory that is gradual, cumulative, and unnoticed until a situation arises that demands the expertise that is no longer there.

The fifth principle is the most demanding and the most important: organizational leaders must decide, explicitly and with full awareness of the trade-offs, what level of human expertise the organization requires and then design its AI deployment to preserve that level. The decision cannot be left to the structural incentives of the market, because the market incentives systematically favor the reduction of human expertise in exchange for the efficiency gains of AI automation. The decision must be made by leaders who understand what Klein's research teaches: that human expertise is not a cost to be minimized but a capability to be maintained, and that the maintenance requires investment that the quarterly earnings cycle does not naturally support.

Klein's research provides the diagnostic framework for making this decision. The RPD model identifies the cognitive functions that expertise serves: pattern recognition, mental simulation, anomaly detection, sensemaking, cognitive flexibility, insight. The expertise paradox identifies the temporal dynamic that makes preservation urgent: the current generation's expertise was built under conditions that AI is eliminating, and the next generation will not develop comparable expertise unless those conditions are deliberately maintained. The trust calibration research identifies the organizational conditions under which human oversight of AI is substantive rather than ceremonial: exposure to failure modes, accurate mental models of system competence, the calibrated skepticism that comes from knowing where the system's boundaries lie.

Together, these concepts constitute something like a field manual for the preservation of human expertise in AI-augmented environments. The manual is not complete. Klein himself would be the first to acknowledge that the AI transition presents challenges that his framework was not designed to address — challenges of scale, speed, and institutional complexity that exceed anything he encountered in his studies of firegrounds and ICUs. But the principles are sound, because they are derived from the most extensive empirical study of human expertise ever conducted, and because the cognitive dynamics they describe — how expertise is built, how it operates, how it degrades — are not artifacts of specific technologies or specific historical periods. They are features of human cognition that persist across contexts.

The fire commander's knowledge was built in a specific way — through direct engagement with a specific domain, over a specific period of time, under specific conditions that produced specific patterns of recognition, simulation, and sensemaking. The knowledge that the AI transition demands will be built in the same way, or it will not be built at all. The technology changes. The cognitive architecture does not. And the organizations that design for the architecture — that invest in the conditions under which human expertise develops and is maintained — will navigate the transition with a resilience that the organizations that optimize only for efficiency will discover, too late, they lack.

Klein once described his research agenda in a sentence that serves as well as any as the organizing principle for the design challenges ahead. He wanted to understand how people could be so smart. Not how they could be so biased, which was Kahneman's question. Not how they could be so rational, which was the classical decision theorist's question. How they could be so smart — how ordinary people, in extraordinary situations, could exercise judgment of a quality and speed that formal models could not match.

The question is no longer just about understanding. It is about preservation. The smartness Klein documented — the fire commander's reading of a structure, the nurse's detection of sepsis, the engineer's feel for a codebase — was built under conditions that are now being transformed at a pace that Klein's career could not have anticipated. The transformation is real, the capabilities it produces are genuine, and the efficiency it delivers is extraordinary. But the smartness that Klein spent a lifetime studying is a non-renewable resource unless the conditions that produce it are deliberately maintained. And maintaining those conditions, in an environment that rewards their elimination, is the design challenge that will define whether the AI transition produces a world in which human intelligence and artificial intelligence complement each other — or one in which the artificial thrives at the expense of the human foundation on which it unknowingly depends.

---

Epilogue

The phrase I cannot stop hearing is one a fire commander said to Gary Klein in 1984. Klein had asked him how he made decisions under pressure. The commander said he did not make decisions. He just knew what to do.

That answer broke open forty years of research, because Klein did not dismiss it. He sat with the impossibility of a man who could not explain his own competence and built an entire science around making the invisible architecture of expertise visible. The patterns that fire commanders cannot articulate. The ten-thousand-hour residue that lets a neonatal nurse detect sepsis before the machines do. The mental simulations that senior engineers run before they commit a line of code — projections so embedded in the act of building that the engineers themselves do not notice they are doing it.

I notice now, after spending months inside Klein's framework, that the ten minutes I described in The Orange Pill — the ten minutes of formative struggle that my engineer in Trivandrum lost when Claude took over the plumbing work — were more important than I understood when I wrote about them. I knew the loss mattered. I could feel it. But Klein gave me the cognitive science to understand why. Those ten minutes were not just practice. They were the raw material from which her pattern library was built, the anomalies that trained her recognition system, the micro-failures that taught her what working code felt like by showing her what broken code felt like. Lose the ten minutes and you lose the deposit. Lose enough deposits and the library thins. Thin the library enough and the engineer who reviews Claude's output can no longer detect the errors that only a thick library catches.

The scariest idea in this book is not that AI will replace experts. It is that AI will produce a generation of practitioners who look like experts — who review output, who approve decisions, who occupy the roles that experts used to occupy — but who lack the experiential foundation that made expertise real. Reviewers without a basis for review. Overseers who have never seen the thing they are supposed to oversee go wrong. Klein calls this the expertise paradox, and it keeps me awake because it describes a failure mode that is invisible in every metric I know how to measure. The dashboards will show green. The outputs will look correct. And underneath, the human infrastructure of judgment that the entire system depends on will be hollowing out, one un-deposited pattern at a time.

What Klein taught me, more than anything, is that the struggle I have been celebrating — the productive friction, the ascending difficulty, the work that gets harder as it gets higher — is not a philosophy. It is a description of how human minds actually build the capacity to be trustworthy in the moments when trust matters most. The fire commander earned his knowing in burning buildings. The nurse earned hers at bedsides. The engineer earned hers in codebases. There is no shortcut, and the removal of the path does not create a better destination. It creates a destination without anyone qualified to know whether they have arrived.

I built Napster Station in thirty days with AI. I am proud of what we built. I also know, now, that the pride means nothing if we do not design the conditions under which the next generation of builders develops the judgment to build things worth being proud of. Klein's framework is the most precise diagnostic I have encountered for understanding what those conditions are, and his career is a four-decade demonstration that the invisible architecture of expertise — the thing that lets a person just know — is worth spending a lifetime trying to see.

The fire commander could not explain his competence. Klein could. That matters now more than it has ever mattered, because we are building systems that depend on a form of human knowing that we are simultaneously making harder to develop. The paradox will not resolve itself. The design must be deliberate. And the first step is seeing what Klein spent his life making visible: that the smartness we are at risk of losing was never effortless. It was earned, pattern by pattern, simulation by simulation, failure by failure, in the slow and irreplaceable currency of direct experience.

The tools are extraordinary. What they amplify had better be real.

— Edo Segal