Herbert Simon

Foreword

By Edo Segal

The decision I thought I was making was never the one I was actually making.

Every time I sat down with Claude during the months I wrote *You On AI*, I believed I was deciding what to build. Architecture, features, prose, product direction. I was choosing among alternatives, picking the best path forward, exercising the judgment that I argue throughout this book is the irreducible human contribution.

What I was not seeing — what I could not see from inside the act — was that the alternatives I chose among had already been filtered. The options Claude presented were a tiny slice of a vast possibility space, selected by criteria I did not set and could not inspect. I was evaluating a curated menu and calling it an open field.

Herbert Simon saw this trap seventy years before I fell into it.

Simon spent his career studying how people actually decide things, and his answer demolished a century of economic theory. We do not optimize. We cannot. Our brains do not have the bandwidth. Instead, we search through options one at a time and grab the first one that clears a bar we call "good enough." He named this *satisficing*, and the concept earned him a Nobel Prize.

But the part that rewired my thinking was not the satisficing itself. It was what happens to that "good enough" bar when the cost of generating the next option drops to zero. The bar rises. It keeps rising. And the cognitive resources you need to evaluate whether each new option actually clears it — your attention, your judgment, your hard-won sense of what matters — those do not rise with it. They stay exactly where biology put them.

This is the asymmetry at the heart of the AI moment: unbounded generation meeting bounded evaluation. Every builder I know feels it. The tools produce faster than you can think. The output looks confident. The temptation to accept without deeply assessing is constant. And the penalty for yielding to that temptation scales with the same multiplier that makes the tools so extraordinary.

Simon gives you the diagnostic framework to see this clearly. Not as philosophy, not as cultural criticism, but as architectural analysis — the precise study of how finite minds should design the systems they inhabit so those systems serve human purposes rather than merely producing human output.

Read this book as an X-ray of the cognitive machinery you bring to every interaction with AI. The machinery has not changed since the Pleistocene. The demands on it have never been greater. Simon understood both facts with a rigor that the current discourse desperately needs.

— Edo Segal ^ Opus

Daniel Kahneman

"Nothing in life is as important as you think it is, while you are thinking about it."

Thinking, Fast and Slow · 2011

→ Daniel Kahneman — On AI: Foreword

Related You On AI Encyclopedia Topics for This Chapter

6 related entries — click to explore the full topic catalog

+

Every one of the 6 Orange Pill Wiki entries this chapter links to — the people, ideas, works, and events it uses as stepping stones. Click any card for the full entry.

Concept (6)

Concept

Attention as Binding Constraint (Simon)

Concept

Bounded Rationality

Concept

Cognitive Exhaustion

Concept

Satisficing

Concept

The Cognitive Revolution

Concept

The Sciences of the Artificial

About Herbert Simon

1916-2001

Herbert Alexander Simon (1916–2001) was an American political scientist, economist, cognitive psychologist, and computer scientist whose work spanned an extraordinary range of disciplines united by a single preoccupation: how human minds actually make decisions under real-world constraints. Born in Milwaukee, Wisconsin, he spent most of his career at Carnegie Mellon University, where he co-founded the field of artificial intelligence alongside Allen Newell and helped establish cognitive science as a rigorous discipline. His concept of *bounded rationality* — the insight that human decision-makers satisfice rather than optimize, operating within the limits of available information, cognitive capacity, and time — earned him the Nobel Memorial Prize in Economic Sciences in 1978 and fundamentally reshaped economics, organizational theory, and the design of institutions. His major works include *Administrative Behavior* (1947), *Organizations* (with James March, 1958), *The Sciences of the Artificial* (1969), and *Human Problem Solving* (with Allen Newell, 1972). Simon also received the A.M. Turing Award in 1975 for his contributions to artificial intelligence, making him one of very few scholars to have earned the highest honors in both economics and computer science. His legacy endures in every field that takes seriously the question of how systems should be designed for the minds that actually inhabit them.

Related You On AI Encyclopedia Topics for This Chapter

1 related entries — click to explore the full topic catalog

+

Every one of the 1 Orange Pill Wiki entries this chapter links to — the people, ideas, works, and events it uses as stepping stones. Click any card for the full entry.

Concept (1)

Concept

The Satisficing Threshold

Chapter 1: The Bounds

In 1955, a political scientist with no formal training in computer science published a paper that dismantled the foundational assumption of modern economics. The assumption was elegant, mathematically tractable, and wrong. It held that human beings, when faced with decisions, gather all available information, evaluate every possible alternative, compute the expected utility of each, and select the option that maximizes their welfare. This model of decision-making — known as rational choice theory — had powered a century of economic thought from Adam Smith through the neoclassical synthesis. It was the bedrock of price theory, game theory, welfare economics, and virtually every formal model of human behavior that had earned its practitioners prestigious appointments.

Herbert Simon demonstrated that the bedrock was sand.

The paper, "A Behavioral Model of Rational Choice," introduced a concept that would earn Simon the Nobel Prize in Economics two decades later: bounded rationality. The argument was not that human beings are irrational. Simon had no patience for that claim, which he considered both empirically lazy and conceptually confused. The argument was more precise and more devastating. Human beings are rational, but their rationality operates within bounds — bounds imposed by the limited information available to them, the limited computational capacity of the human brain, and the limited time in which decisions must be made. Given these constraints, human beings do not optimize. They cannot. The informational requirements of optimization exceed the cognitive resources of any biological mind. What they do instead is satisfice: they search through available alternatives until they find one that meets a minimum threshold of acceptability, and then they stop searching.

The distinction between optimizing and satisficing is not a matter of degree. It is a difference in kind. The optimizer evaluates all alternatives and selects the best. The satisficer evaluates alternatives sequentially and selects the first one that is good enough. The optimizer needs complete information, unlimited computation, and infinite time. The satisficer needs only a threshold — a standard of acceptability — and the capacity to evaluate alternatives one at a time until the threshold is met.

Amos Tversky

"It's frightening to think that you might not know something, but more frightening to think that, by and large, the world is run by people who have faith that they know exactly what is going on."

Interview, ca. 1990s ·

→ Amos Tversky — On AI: About

Simon drew the evidence from everywhere a political scientist who had wandered into economics, psychology, and computer science could draw it. Corporate executives did not analyze every possible strategic direction before choosing one. They considered a handful of plausible options and selected the first that met their board's minimum criteria. Chess grandmasters did not evaluate every legal move on the board. They considered four or five moves — selected by pattern-recognition heuristics built from thousands of hours of practice — and chose from among those. Municipal administrators did not calculate optimal budget allocations. They used rules of thumb inherited from predecessors and adjusted at the margins. In every domain Simon examined, the same pattern held: bounded agents using heuristics to navigate environments too complex for exhaustive analysis.

The insight was not merely descriptive. It was architectural. If human rationality is bounded, then every institution, every organization, every decision-making structure ever built by human beings is, at bottom, a device for managing those bounds. The corporate hierarchy is not an expression of power for its own sake. It is a decomposition architecture — a structure that breaks decisions too complex for any single mind into sub-decisions manageable by bounded minds working in coordination. The division of labor is not merely an efficiency mechanism. It is a cognitive strategy — a way of ensuring that no single decision-maker needs to hold more information than bounded rationality can process. The standard operating procedure, the committee, the reporting chain, the spec document, the sequential handoff between teams — all are prosthetics for cognitive limitation. They exist not because they represent the best way to organize work, but because they represent the way that is compatible with the bounds of human cognition.

This architectural insight is what makes bounded rationality more than a footnote in the history of economic thought. Classical economics described a species that does not exist — Homo economicus, the perfectly rational calculator — and built an entire edifice of theory on that fiction. Simon described the species that does exist and asked: given what human minds can actually do, how should the systems around them be designed?

The question was radical because it dissolved the boundary between description and prescription, between how things are and how they should be built. Simon spent the rest of his career in that dissolved space, moving between cognitive psychology, organizational theory, computer science, and the field he helped invent — artificial intelligence — always asking the same underlying question: given bounded rationality, what structures best serve the agents who inhabit them?

Nearly seventy years after that 1955 paper, a technology arrived that appeared to dissolve the bounds themselves.

The builder who works with Claude Code in 2026 operates under conditions that classical economics imagined but Simon proved impossible for biological minds. The informational constraint — the limited data available to any single decision-maker — has been relaxed to a degree that would have astonished Simon, who spent the last decades of his life studying how information technology would reshape organizational cognition. Claude has been trained on a substantial fraction of the publicly available text produced by human civilization. The builder who converses with it has access, through that conversation, to a reservoir of information that no individual, no team, no organization in human history has previously commanded.

The computational constraint has been similarly relaxed. The builder who describes a software system to Claude does not need to hold the implementation details in working memory — the roughly four to seven chunks of information that cognitive psychology identifies as the upper bound of human processing capacity at any given moment. The machine holds the implementation. The builder holds the intention. The computational bottleneck that forced every previous generation of builders to choose between breadth and depth — you could understand the whole system shallowly or one component deeply, but not the whole system deeply, because the cognitive resources did not exist — has been restructured. The machine provides computational depth across the full breadth of the system. The builder provides direction.

The time constraint has been compressed beyond anything Simon's framework anticipated. When implementation cost drops to the duration of a conversation, the sequential search that bounded rationality imposes — evaluate one alternative, then the next, then the next, stopping at the first acceptable result — can be replaced by something approaching parallel evaluation. The builder can describe three approaches in the time it would have taken to implement one. The satisficing calculus changes when the cost of generating the next alternative approaches zero.

From the outside, this looks like unbinding. The bounds that produced Simon's theory — information limits, computational limits, time limits — appear to have been removed. The builder who works with AI appears to operate in the conditions that classical economics assumed and Simon proved fictional: complete information, unlimited computation, no meaningful time constraint on the generation of alternatives. If the bounds have been removed, then bounded rationality is an artifact of a previous technological era, and the organizational structures it produced — the hierarchies, the handoffs, the divisions of labor — are obsolete.

Simon's framework predicts otherwise. The prediction rests on a distinction that the current discourse about AI has largely failed to make — the distinction between the bounds that have been relaxed and the bounds that remain intact.

Three constraints produced bounded rationality: limited information, limited computation, and limited time. AI has relaxed all three, though not to infinity. But Simon identified a fourth constraint that operates independently of the other three and that AI has not relaxed at all. In 1971, writing about the design of information systems for organizations, Simon stated the problem with a precision that reads, half a century later, as prophecy: "What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information sources that might consume it."

Attention. Not information, not computation, not time — attention. The cognitive resource required to evaluate what the information means, to judge whether the computation has produced the right answer, to decide whether the time saved should be invested in generating another alternative or accepting the current one.

AI has expanded the first three constraints while leaving the fourth untouched. The builder who works with Claude has access to more information, more computational power, and more speed than any human being in history. Her capacity to evaluate what all of that information, computation, and speed has produced — to direct her attention toward the outputs that matter and away from the outputs that do not — remains as bounded as it was before the machine arrived.

The bounds have not been removed. They have been relocated.

In the old world, the binding constraint was generation: the builder could not produce alternatives fast enough. Information was scarce, computation was expensive, implementation took months. Bounded rationality expressed itself as a limitation on what could be created. The satisficing threshold was low — not because standards were low, but because the cost of generating the next alternative was high. The builder accepted the third option not because it was good enough in some absolute sense, but because the fourth option would have taken another six weeks to produce.

In the new world, the binding constraint is evaluation: the builder cannot assess alternatives wisely enough. Information is abundant. Computation is nearly free. Implementation takes hours. The generation bottleneck has been obliterated. But the builder's capacity to judge the quality of what has been generated — to distinguish the elegant from the merely functional, the robust from the fragile, the solution that serves the user from the solution that merely satisfies the prompt — has not expanded proportionally. The evaluative bottleneck remains. It has, in fact, become more severe, because the volume of material demanding evaluation has increased by orders of magnitude while the cognitive apparatus doing the evaluating has not changed since the Pleistocene.

This asymmetry — unbounded generation meeting bounded evaluation — is the central cognitive fact of the AI age. It is what produces the specific exhaustion that researchers at UC Berkeley documented in 2026, when they embedded themselves in a technology company and found that AI did not reduce work but intensified it. The workers generated more, evaluated more, took on wider scope, and burned out faster. The satisficing threshold had risen — the cost of generating another alternative had dropped, so "good enough" now required evaluating more alternatives before stopping — but the cognitive resources available for evaluation had not risen with it.

Simon's framework does not merely predict this asymmetry. It explains why the asymmetry will persist regardless of how powerful the tools become. Because attention is not an information problem. It is not a computation problem. It is not a speed problem. It is a consciousness problem — a problem inherent in the condition of being a finite mind that can hold only a finite number of things in awareness simultaneously. No tool can expand this bound without fundamentally altering what it means to be a conscious agent. The builder who works with the most powerful AI system imaginable will still need to decide where to direct her attention, and that decision will still be bounded by the same cognitive architecture that bounded the decisions of Simon's municipal administrators in 1947.

The implications of this relocation ripple outward into every domain that You On AI addresses. The organizational structures that Segal describes dissolving — the specialist silos, the sequential handoffs, the twenty-person teams — were architectures designed for the old bounds. They decomposed problems into generation-sized chunks because generation was the bottleneck. When the bottleneck shifts to evaluation, the architecture must shift with it. The new organizational question is not "how do we decompose the work of building?" but "how do we support the work of judging what should be built?"

George Miller

"The span of absolute judgment and the span of immediate memory impose severe limitations on the amount of information that we are able to receive, process, and remember."

The Magical Number Seven, Plus or Minus Two · 1956

→ George Miller — On AI: Chapter 1

The educational structures require the same rethinking. The pedagogy that trained builders to generate — to write code, to draft briefs, to build models — served a world where generation was the scarce skill. In a world where generation is cheap and evaluation is scarce, the pedagogy must shift toward building the cognitive capacities that evaluation demands: judgment, taste, the ability to recognize when something is wrong before being able to articulate what, the meta-cognitive awareness to ask whether a confident answer is actually a correct one.

The personal practices that individuals develop for managing their own cognitive resources become, in Simon's framework, the most consequential design decisions any builder makes. How attention is allocated — what gets evaluated carefully and what gets accepted on the AI's authority — determines the quality of everything downstream. This allocation is itself a satisficing decision, made under bounded rationality, subject to the same heuristics and biases that Simon spent his career cataloging.

The bounds have shifted. The architecture designed for the old bounds is being dismantled. The question — the Simonian question, the design question, the only question that matters — is whether new architectures will be built to serve the bounds that remain. The answer will determine whether the AI age produces builders of extraordinary capability or builders of extraordinary output who cannot tell the difference between the two.

Related You On AI Encyclopedia Topics for This Chapter

12 related entries — click to explore the full topic catalog

+

Every one of the 12 Orange Pill Wiki entries this chapter links to — the people, ideas, works, and events it uses as stepping stones. Click any card for the full entry.

Concept (12)

Concept

Administrative Behavior

Concept

Attentional Ecology

Concept

Chunking (Miller)

Concept

Cognitive Offloading

Concept

Deep Work

Concept

Designing for the Bounded Builder

Concept

Engagement Optimization

Concept

Metacognition — Thinking About Thinking

Concept

Prospect Theory

Concept

Sociotechnical System

Concept

The Coordination Bottleneck

Concept

The Relocation of Bounds

Chapter 2: The Satisficing Threshold

The word did not exist before Simon invented it. "Satisfice" — a portmanteau of "satisfy" and "suffice" — entered the vocabulary of the social sciences in 1956, in a paper called "Rational Choice and the Structure of the Environment." The neologism was necessary because the concept it described had no name, despite being the most common form of human decision-making. Optimizing had a name. Maximizing had a name. The thing that human beings actually do when they face complex decisions in real environments had no name at all, because the discipline that studied decision-making had been so thoroughly captured by the fiction of optimization that the reality of how decisions actually get made had been rendered invisible.

Satisficing is a search procedure. The satisficer faces a set of alternatives that she cannot evaluate simultaneously — the set is too large, the evaluation too costly, the time too short. Instead of evaluating all alternatives and selecting the best, she evaluates them sequentially. One at a time. Each alternative is compared not to every other alternative but to a threshold — a minimum standard of acceptability that the decision-maker carries, often implicitly, as a criterion for "good enough." When an alternative meets or exceeds the threshold, the search terminates. The satisficer selects that alternative and moves on.

The threshold is not fixed. It adjusts. When acceptable alternatives are easy to find — when the environment is rich and the search productive — the threshold drifts upward. The satisficer becomes more demanding because the cost of continuing to search is low relative to the probability of finding something better. When acceptable alternatives are hard to find — when the environment is sparse and the search frustrating — the threshold drifts downward. The satisficer becomes less demanding, accepting alternatives she would have rejected in a richer environment, because the cost of continuing to search exceeds the expected value of the improvement.

This dynamic — the threshold adjusting to the cost of search — is the mechanism that connects satisficing theory to every major technological transition in the history of human tool use. Each transition that reduced the cost of generating alternatives shifted the satisficing threshold upward. Each upward shift produced both better outcomes and greater cognitive demand on the decision-maker.

The printing press reduced the cost of producing text. Before Gutenberg, a manuscript required months of scribal labor. After Gutenberg, a text could be reproduced in days. The satisficing threshold for publishers — the minimum quality of a text worth producing — shifted upward. When producing the next book cost a fraction of what producing the last book had cost, publishers could afford to be more selective. They could reject manuscripts they would have accepted when the production cost was higher, because the cost of evaluating the next candidate had dropped along with the cost of producing it. The result was more books, of higher average quality, requiring more evaluative labor from the people who decided which books deserved to exist. The publisher's role expanded from curator to gatekeeper, a role that did not need to exist when the production bottleneck was so severe that only works of obvious merit could justify the scribal investment.

The spreadsheet reduced the cost of producing financial models. Before VisiCalc, a financial projection required days of manual calculation. After VisiCalc, it required hours. The satisficing threshold for financial analysts shifted upward. When the next scenario could be modeled in minutes rather than days, analysts could afford to reject projections they would have accepted in the manual era. They evaluated more scenarios before declaring the analysis complete. The result was more thorough financial analysis and more exhausted financial analysts — the same asymmetry that recurs at every transition, because the cost of generation dropped while the cost of evaluation remained constant.

The word processor reduced the cost of revising text. Before word processing, revision meant retyping the entire manuscript. After word processing, revision meant editing in place. The satisficing threshold for writers shifted upward. When the next draft cost minutes rather than hours, writers revised more before accepting a manuscript as finished. The result was, on average, more polished prose — and the specific neurosis of the writer who cannot stop revising, because the cost of one more pass has dropped below the threshold at which stopping feels justified. The optimizer's trap, dressed in satisficing clothing.

AI accelerates this dynamic to a qualitatively new regime.

When the cost of generating a working software prototype drops from months to minutes — when a builder can describe a system in natural language and receive a functional implementation in the time it takes to have a conversation — the satisficing threshold does not merely shift upward. It undergoes a phase transition. The economics of search are so radically altered that the behavioral patterns characteristic of a world where generation is expensive become incoherent. The builder who would have stopped at the third approach because the fourth would have taken six weeks to implement now has no external reason to stop at any number. The fourth approach costs ten minutes. So does the fifth. The tenth. The hundredth.

The external cost of search has approached zero. The internal cost — the cognitive demand of evaluating each alternative against the builder's criteria for quality, feasibility, elegance, and fit — has not changed at all. Evaluation remains as expensive as it was before the machine arrived. Each alternative still demands attention. Each still requires the exercise of judgment — the integration of multiple criteria that cannot be reduced to a formula and must be performed by a mind that can hold the relevant considerations simultaneously. The builder's evaluative capacity is the same four-to-seven chunks of working memory, the same pattern-recognition heuristics built from years of experience, the same aesthetic and practical intuitions that constitute what builders call "taste."

The result is a specific kind of cognitive trap that Simon's framework predicts with uncomfortable precision. The builder generates alternatives at machine speed. She evaluates them at human speed. The gap between generation and evaluation widens with each iteration. The satisficing threshold, which adjusts to the cost of search, keeps rising — because the cost of generating the next alternative keeps falling — but the evaluative resources available to determine whether the new alternative actually exceeds the threshold do not rise with it. The builder finds herself in a regime where the standard for "good enough" is perpetually one alternative ahead of her capacity to assess whether "good enough" has been reached.

This is the mechanism beneath the surface of the behavioral patterns that researchers at UC Berkeley documented. The workers in their study were not being exploited by external taskmasters. They were being driven by the internal logic of satisficing in a low-cost-generation environment. When producing the next deliverable costs almost nothing, the threshold for "enough work done today" recedes like a horizon. The worker generates another feature, another draft, another analysis — not because any external authority demands it, but because the satisficing calculus, operating as it always has, registers that the cost of the next attempt is trivially low relative to the potential value of an improvement. The search continues. The attention depletes. The threshold keeps rising.

Byung-Chul Han, the philosopher whose diagnosis of the "achievement society" figures prominently in You On AI, describes this pattern as self-exploitation — the internalization of the productivity imperative to the point where the worker becomes her own overseer. Simon's framework offers a more precise mechanism. The worker is not exploiting herself. She is satisficing on a cost curve that has been radically reshaped by AI, and the rational response to the new cost curve is to search longer, evaluate more, and stop later than the old cost curve required. The exhaustion is not irrational. It is the predictable outcome of rational satisficing in an environment where the cost of generation has collapsed while the cost of evaluation has not.

The distinction matters because it implies different interventions. If the problem is self-exploitation — a cultural pathology of internalized achievement pressure — then the solution is cultural: different values, different metrics, a philosophical reorientation away from productivity and toward contemplation. This is Han's prescription, and it has a certain purity that Simon's framework admires in the abstract and finds unimplementable in practice. Human beings are not going to stop satisficing. They are not going to stop adjusting their thresholds to the cost of search. These are not cultural artifacts. They are cognitive mechanisms — features of the decision-making architecture that evolution built and that no philosophical reorientation will undo.

If the problem is, instead, a misaligned satisficing threshold — a cost curve that drives rational search past the point of diminishing returns — then the solution is architectural. Redesign the environment in which the satisficing decision is made. Build structures that impose evaluation costs on the search process: mandatory reflection periods, peer review gates, protected time for the kind of slow assessment that the machine's speed makes feel unnecessary. Not because contemplation is inherently virtuous — Simon had little patience for arguments that dressed up preferences as principles — but because the quality of the output depends on the quality of the evaluation, and evaluation quality degrades when the evaluator is operating under the attentional exhaustion produced by an unrestricted search.

The most effective satisficing boundaries are not the ones the builder imposes on herself through willpower. Willpower is itself a bounded resource, and deploying it against the grain of a cost curve that makes continued search feel rational is a losing strategy. The most effective boundaries are structural — built into the environment so that the satisficing decision encounters resistance at the appropriate point.

Simon studied these structural boundaries in organizations for decades. The committee that must approve a decision before implementation is a satisficing boundary — it imposes an evaluation cost that prevents the decision-maker from acting on the first alternative that exceeds her individual threshold. The budget cycle that forces resource allocation decisions at fixed intervals is a satisficing boundary — it imposes a temporal structure that terminates search regardless of whether the decision-maker feels the search is complete. The peer review process in science is a satisficing boundary — it interposes an external evaluative standard between the researcher's judgment that a finding is "good enough" and the finding's entry into the public record.

Each of these structures has costs. They slow things down. They frustrate the builder who wants to move at the speed the tool makes possible. They are friction, and the dominant aesthetic of the technology industry — the aesthetic of the smooth, as Han would call it — treats friction as the enemy. But Simon's framework reveals that friction is not the enemy. Undirected search is the enemy. Friction is the mechanism by which search is directed — the structure that channels the builder's bounded evaluative resources toward the alternatives most worth evaluating and away from the alternatives that the machine can generate endlessly but that no bounded mind can assess.

The Trivandrum training that You On AI describes — twenty engineers achieving what Segal calls a twenty-fold productivity multiplier — is, in Simonian terms, a story about a satisficing threshold that shifted dramatically upward in the space of five days. The engineers were suddenly capable of generating alternatives at a rate their previous experience had not prepared them for. The question the account raises but does not fully answer is what happened to the quality of their evaluation during those five days. Was the evaluation keeping pace with the generation? Were the twenty-fold-more-productive engineers also evaluating their output twenty times more carefully? Or were they generating at machine speed while evaluating at human speed, satisficing at a threshold that the excitement of the moment had temporarily inflated beyond what their bounded evaluative capacities could sustain?

The answer, from Simon's framework, is predictable: the evaluation was not keeping pace. It could not. The evaluative capacity is bounded by the same cognitive architecture whether the engineer is generating one alternative per week or twenty per day. The likely outcome is that the most capable engineers — those with the deepest pattern libraries, the richest meta-heuristic repertoires, the most refined sense of what "good enough" actually means in their domain — were able to maintain evaluation quality at the higher generation rate, because their heuristics were powerful enough to assess alternatives rapidly. The least experienced engineers — those whose heuristic libraries were thinnest, whose pattern-recognition was least developed — were most likely to let evaluation quality slip, satisficing on outputs that met a surface-level threshold but concealed the kind of structural weaknesses that only experienced evaluators can detect.

The implication is uncomfortable for the democratization narrative. AI democratizes generation. It does not democratize evaluation. The experienced engineer and the novice now have access to the same generative power. They do not have access to the same evaluative power. The asymmetry between generation and evaluation — already present in every previous technological transition — is wider in the AI age than it has ever been, because the generation side has been expanded by orders of magnitude while the evaluation side has remained fixed.

This asymmetry is not a temporary problem awaiting a technical solution. It is a permanent feature of any system in which bounded minds interact with unbounded generation. The satisficing threshold will continue to rise. The evaluative bottleneck will continue to bind. The quality of the output will continue to depend not on the power of the tool but on the wisdom of the mind directing it — wisdom that can only be built through the slow, expensive, irreducibly human process of accumulating judgment through experience, failure, and reflection.

Simon knew this. He spent the last decades of his life studying expertise and how it is built. The answer he found — roughly ten years of deliberate practice, resulting in fifty thousand to a hundred thousand chunks of pattern-recognition knowledge — did not lend itself to acceleration. The mechanism that builds expert judgment is not amenable to the same compression that AI brings to generation. Judgment accretes. It deposits in layers. Each layer requires the specific resistance of having encountered a problem, attempted a solution, observed the failure, and integrated the lesson. The process is slow because the architecture of human learning is slow — not because previous tools were inadequate, but because the biological substrate on which learning occurs has its own temporal requirements.

The satisficing threshold has risen. The tool that would help it rise wisely — the tool that would expand evaluative capacity at the same rate it has expanded generative capacity — does not exist and may not be buildable within the constraints of human cognition. What can be built are structures that respect the asymmetry: organizational designs, educational practices, and personal disciplines that impose appropriate boundaries on search, that direct bounded evaluative attention toward the alternatives most worth assessing, and that resist the seductive logic of a cost curve that says the next attempt is free and therefore mandatory.

Douglas Engelbart

"The complexity of problems we can address is largely determined by our ability to augment the collective IQ of human organizations."

Augmenting Human Intellect: A Conceptual Framework · 1962

→ Douglas Engelbart — On AI: Chapter 2

Related You On AI Encyclopedia Topics for This Chapter

12 related entries — click to explore the full topic catalog

+

Every one of the 12 Orange Pill Wiki entries this chapter links to — the people, ideas, works, and events it uses as stepping stones. Click any card for the full entry.

Concept (12)

Concept

Ascending Friction

Concept

Attention Residue

Concept

Attentional Agreements

Concept

Cognitive Flexibility (Klein)

Concept

Cognitive Tools

Concept

Deliberation

Concept

Mental Models

Concept

Self-Efficacy

Concept

The Burnout Society

Concept

The Capability Trap

Concept

The Framing Effect

Concept

The Judgment Bottleneck

Chapter 3: Attention as the Scarce Resource

Half a century before the phrase "attention economy" entered common usage, before social media platforms discovered that human attention could be harvested, measured, and sold to advertisers at scale, before the behavioral science of engagement optimization produced the notification systems and infinite scroll mechanisms that now saturate daily life, Herbert Simon identified the binding constraint of the information age with a precision that subsequent decades have only confirmed. Writing in 1971, in a paper for a symposium on the economics of information processing, he stated the problem in two sentences that now read less as academic observation than as prophecy fulfilled: "What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information sources that might consume it."

The formulation is worth pausing over, because its structure reveals the logic that makes it predictive rather than merely descriptive. Simon does not say that information competes with attention, as though the two were independent forces that happen to conflict. He says information consumes attention — the relationship is metabolic, not competitive. Information is the fuel. Attention is what burns. More fuel does not produce a larger fire if the furnace is the same size. It produces waste — unburned fuel accumulating around a furnace operating at maximum capacity, generating heat but also smoke, requiring ever more sophisticated ventilation systems to prevent the entire structure from choking on its own abundance.

The metaphor maps with uncomfortable directness onto the condition of the AI-augmented builder. The furnace is the builder's cognitive apparatus — the roughly four to seven chunks of working memory, the pattern-recognition systems built from years of domain experience, the evaluative heuristics that constitute professional judgment. The fuel is the output of the AI system — code, documentation, alternatives, approaches, connections, possibilities, each one requiring attention to evaluate, each one consuming a slice of the finite cognitive resource that the builder brings to the interaction. The AI generates fuel without limit. The furnace remains the same size it has been since the Pleistocene.

Simon did not merely diagnose the problem. He prescribed the design principle that follows from it: information systems should serve as attention conservers. Not information maximizers — the instinct of every system designed in ignorance of bounded rationality — but attention conservers. The function of a well-designed information system, in Simon's framework, is not to deliver the maximum amount of information to the decision-maker. It is to deliver the minimum amount of information necessary for the decision at hand, filtering everything else before it reaches the decision-maker's attention. The system should be, in Simon's precise phrase, "an information condenser" — a mechanism that compresses the vast space of available information into the small space that bounded attention can productively engage.

The history of information technology since 1971 has been, with remarkable consistency, a history of systems designed in violation of this principle. The email inbox delivers every message to the recipient's attention regardless of its importance, relevance, or urgency. The social media feed optimizes for engagement — for the consumption of attention — rather than for the conservation of it. The search engine returns thousands of results ranked by relevance algorithms that approximate the user's intent but cannot substitute for the user's judgment about which result actually serves her purpose. Each system expands the information available to the decision-maker. None of them conserve the decision-maker's attention.

AI tools enter this landscape as both the most powerful information expanders ever built and, in principle, the most powerful attention conservers ever imagined. The duality is not accidental. It follows from the same capability. A system that can process natural language, generate code, summarize documents, identify patterns, and produce structured output from unstructured input has the capacity to function as either a firehose or a filter — to either flood the builder with possibilities or curate those possibilities into a manageable set that respects the builder's attentional bounds.

Which function it serves depends not on the tool itself but on how the tool is used, how the interaction is structured, and whether the builder — or the organization, or the educational institution — has designed the interaction with attention conservation as an explicit goal. The default, absent deliberate design, is expansion. The AI will generate everything it can. The builder will attempt to evaluate everything generated. The attention deficit will compound.

Consider the specific attentional economics of a builder working with Claude Code on a complex software system. She describes a feature. Claude produces an implementation. The implementation works — it passes the tests, handles the edge cases, runs without errors. The builder can accept it and move on. Or she can ask for an alternative approach. Claude produces a second implementation, different in architecture, different in tradeoffs, potentially better in ways the builder cannot assess without careful examination. She can accept this one. Or she can ask for a third.

Each alternative costs the builder attention. Not the superficial attention of glancing at the output — the deep attention required to understand the architectural choices, evaluate the tradeoffs, assess the implications for the broader system, consider the maintenance burden, anticipate the failure modes. This is the kind of attention that Simon studied in chess grandmasters: not the attention required to see the board but the attention required to understand the position, which involves activating pattern-recognition systems that compare the current state against thousands of previously encountered states and generate an assessment that feels, to the practitioner, like intuition but is in fact the rapid retrieval and comparison of deeply encoded knowledge.

The builder who evaluates a single implementation with full attention exercises this kind of deep assessment. The builder who evaluates five implementations in the same time period cannot exercise it five times as deeply. Attention does not parallelize. The cognitive architecture that produces deep evaluation is serial — it processes one thing at a time, holding it in the focus of consciousness long enough for the pattern-recognition systems to complete their work. Five implementations evaluated in the time previously allocated to one means each implementation receives one-fifth the attentional investment. The satisficing threshold may have risen — the builder rejects alternatives she would have accepted in a lower-generation environment — but the quality of each evaluation has declined proportionally.

This is what Simon's framework predicts and what the empirical evidence from AI-augmented workplaces confirms: the shift from generation scarcity to generation abundance does not produce a proportional improvement in decision quality. It produces a reallocation of attention — from deep evaluation of few alternatives to shallow evaluation of many — and the net effect on decision quality depends on whether the additional alternatives compensate for the reduced depth of evaluation per alternative.

In some cases, they do. When the builder's initial implementation would have contained a critical flaw that any of the four alternative implementations would have avoided, the shallow-but-wide evaluation strategy produces a better outcome than the deep-but-narrow strategy. The fourth alternative, even cursorily evaluated, is better than the first alternative, deeply evaluated, if the first alternative was fundamentally wrong.

But in many cases, they do not. When the differences between alternatives are subtle — when the choice between them depends on tradeoffs that require deep domain knowledge to assess, when the failure modes are invisible at surface level, when the implications for the broader system unfold over months rather than minutes — then the shallow evaluation of many alternatives produces worse decisions than the deep evaluation of few. The builder selects an alternative that looks correct at the level of attention she has allocated to it, but that conceals a structural weakness discoverable only through the kind of sustained, concentrated analysis that the abundance of alternatives has crowded out.

The Deleuze episode that Segal recounts in You On AI is an instance of this failure mode operating at the level of intellectual rather than software architecture. Claude produced a passage connecting Csikszentmihalyi's flow state to a concept attributed to Gilles Deleuze. The passage was eloquent, structurally sound, and rhetorically effective. It consumed a small amount of evaluative attention — enough for the author to register that it "sounded right" and "felt like insight." It did not consume enough evaluative attention for the author to verify the philosophical reference against his own knowledge of Deleuze's actual work. The attention required for that verification — the retrieval of specific knowledge about Deleuze's concept of smooth space, the comparison of that concept against how Claude had deployed it, the assessment of whether the deployment was legitimate — was crowded out by the volume of other material demanding evaluation simultaneously.

The error was caught the next morning, not through deliberate re-evaluation but through the kind of nagging pattern-mismatch that experienced minds register below the threshold of conscious attention and surface later, often during periods of low cognitive load. This is precisely the mechanism that Simon's collaborator William Chase documented in chess expertise research: the expert's pattern library generates a low-level alert when something does not match the stored patterns, even before the expert can articulate what the mismatch is. But this mechanism requires two conditions: a deep pattern library, built through years of domain exposure, and sufficient attentional slack for the low-level alert to surface into consciousness. AI-augmented work threatens both conditions — the first by potentially reducing the experiential depth that builds the library, the second by filling every available cognitive space with material demanding evaluation.

Simon extended his attention analysis beyond individual cognition to organizational design. If attention is the scarce resource, then the primary function of organizational structure is not to coordinate the flow of work but to manage the allocation of attention. The hierarchy, in Simon's framework, is an attention architecture — a structure that determines who attends to what, at what level of detail, and under what conditions information should be escalated from one attentional level to another.

The junior analyst attends to the data. The senior manager attends to the patterns in the data. The executive attends to the patterns in the patterns. Each level of the hierarchy filters information, condensing it before passing it upward, so that each successive decision-maker faces a smaller, more manageable set of alternatives requiring the kind of deep attention that bounded cognition can productively exercise. The hierarchy is, in Simon's precise language, a cascading attention filter — each level conserving the attention of the level above it by absorbing and processing the information that would otherwise overwhelm it.

Norbert Wiener

"We are not the stuff that abides, but patterns that perpetuate themselves."

The Human Use of Human Beings · 1950

→ Norbert Wiener — On AI: Chapter 3

AI disrupts this architecture by enabling individuals to operate across levels. The builder who works with Claude Code can attend simultaneously to implementation details and architectural patterns — domains that the traditional hierarchy allocated to different people. The organizational hierarchy was designed for an era where each person's attention could encompass only their level. When the tool expands the range of what one person can hold in view, the levels compress. The junior analyst, augmented by AI, produces output that competes for the senior manager's attention alongside the output of other analysts and the manager's own analysis. The information-condensation function that the hierarchy performed has been disrupted without being replaced.

The result is predictable from Simon's framework: attentional overload at the upper levels of the organization. The decision-makers whose attention was previously protected by the filtering hierarchy now face a volume of high-quality output — generated at every level, by every AI-augmented individual — that exceeds their evaluative capacity. The hierarchy was a dam. The dam has been breached. The water — the information, the alternatives, the possibilities — flows upward unfiltered, and the executives at the top of the structure are drowning in material that is individually excellent and collectively unprocessable.

The prescription, again, is architectural. Not "use less AI" — that horse has left the barn and crossed the county line. The prescription is: redesign the organizational attention architecture for the new information environment. Build new filtering mechanisms that perform the condensation function the old hierarchy performed. Create evaluation structures — the "vector pods" that Segal describes, the AI Practice frameworks that the Berkeley researchers recommend — that impose attentional discipline on a system that, left to its default dynamics, will consume every unit of attention available to it and still demand more.

Simon's 1971 formulation — that information consumes attention, and that the design challenge is to conserve attention rather than maximize information — has not been superseded by the arrival of AI. It has been vindicated by it. The principle was always true. The consequences of ignoring it were always present. AI has merely amplified those consequences to the point where they can no longer be absorbed by the informal coping mechanisms — the coffee break, the closed office door, the weekend — that previously managed the attention deficit at a level too low to demand formal architectural response.

The formal response is now overdue. The attention architecture of the AI age remains, in most organizations, undesigned — an absence that Simon's framework identifies as the single most dangerous feature of the current moment. Not the power of the tools. Not the speed of the transformation. The absence of structures designed to conserve the one resource that determines whether everything else produces value or merely produces volume.

Related You On AI Encyclopedia Topics for This Chapter

13 related entries — click to explore the full topic catalog

+

Every one of the 13 Orange Pill Wiki entries this chapter links to — the people, ideas, works, and events it uses as stepping stones. Click any card for the full entry.

Concept (13)

Concept

Abstraction and Professional Control

Concept

Attentional Apnea

Concept

Behavioral Surplus

Concept

Cognitive Ease

Concept

Cognitive Exhaustion

Concept

Cognitive Flexibility (Klein)

Concept

Cognitive Maps

Concept

Creative Destruction

Concept

Deep Work

Concept

Information Asymmetry

Concept

The Attentional Commons

Concept

The Creative Director in the AI Economy

Concept

The Filter Bubble

Chapter 4: The Architecture of Choice

In the late 1940s, before the terms "user interface" or "user experience" existed, before the discipline of design thinking had been formalized, before a generation of behavioral economists would build careers on the insight that the structure of a decision environment shapes the decisions made within it, Herbert Simon was studying how administrative organizations made decisions. What he found was that the decisions made by individuals within organizations were determined less by the preferences of the decision-makers than by the structure of the decision environment in which they operated. The options available, the information highlighted, the defaults in place, the sequence in which alternatives were presented — these architectural features of the decision context shaped outcomes more reliably than individual rationality, individual preference, or individual skill.

The finding was counterintuitive. The prevailing model assumed that a rational agent's decisions reflect the agent's preferences, modified by the information available. Simon demonstrated that this model inverts the actual causal chain. A bounded agent's decisions reflect the decision architecture, modified by the agent's preferences. The architecture comes first. The preferences operate within it — amplified, constrained, redirected, and sometimes overridden by the structure of the environment in which the choice is made.

Consider a simple example that Simon studied in depth: the routing of information within a bureaucracy. A municipal administrator receives reports from several departments. The reports arrive on her desk in an order determined not by importance but by the timing of departmental reporting cycles. The administrator reads them in the order received. She attends most carefully to the first report — the one that arrives when her attention is freshest — and least carefully to the last. Her decisions about resource allocation reflect this attentional gradient. Departments whose reports arrive early receive more careful consideration than departments whose reports arrive late, regardless of the relative importance of the issues involved.

The administrator is not irrational. She is boundedly rational, operating within a decision architecture that presents information in a particular order, and her bounded attention — the scarce resource — is allocated according to that presentation. If the architecture changes — if the reports are reordered by importance rather than by arrival time — her decisions change. Not because her preferences have changed, but because the architecture that channels her bounded attention has been redesigned.

This insight — that the structure of the decision environment is often more consequential than the quality of the decision-maker — was decades ahead of its theoretical consolidation. It would not be fully articulated until Thaler and Sunstein's Nudge in 2008, which translated Simon's organizational observations into a popular framework for "choice architecture" — the deliberate design of decision environments to channel bounded rationality toward better outcomes. But the core mechanism was Simon's: bounded minds make decisions within architectures, and the quality of the decision is a joint product of the mind and the architecture. Change the architecture, and you change the decision, even when the mind remains the same.

AI is a choice architecture. This claim is less obvious than it might appear, because the dominant framing of AI tools positions them as neutral instruments — tools that execute the builder's intentions, amplifiers that carry whatever signal the builder provides. The amplifier metaphor, which You On AI employs to powerful effect, captures something real about the relationship between builder and tool. But it obscures something equally real: the tool does not merely amplify the builder's signal. It structures the environment in which the builder's signal is formed, shaped, and expressed.

When a builder describes a problem to Claude, the AI's response creates a decision environment. The response presents certain alternatives and not others. It foregrounds certain approaches and backgrounds others. It structures the problem in a particular way — decomposing it into components, sequencing the components, proposing a solution architecture — and this structuring is not neutral. It reflects the patterns in the training data, the architectural preferences encoded in the model's weights, the interpretation of the builder's intent that the model's inference process has generated. The builder then makes decisions within this structured environment. She accepts, modifies, or rejects the alternatives presented. She evaluates the approach proposed. She judges the quality of the output.

But her evaluation, her judgment, her acceptance or rejection, all operate within the frame that the AI has constructed. The alternatives she did not see — the approaches the AI did not propose, the decompositions the AI did not consider, the solutions outside the distribution of the training data — are invisible. They are not rejected. They are simply absent from the decision environment. And absence, in a choice architecture, is the most powerful form of influence, because it operates below the threshold of awareness. The builder cannot evaluate what she does not see. She cannot choose what has not been presented. The architecture shapes the decision by determining the boundaries of the space within which the decision is made.

This is not a deficiency of any particular AI system. It is a structural feature of any system that mediates between a bounded mind and a vast possibility space. The possibility space of software architecture, of prose composition, of product design, of any complex creative domain, is too large for any mind — bounded or unbounded — to search exhaustively. The AI, like the organizational hierarchy Simon studied, performs a filtering function. It condenses the vast space into a manageable set of alternatives that it presents to the builder for evaluation. The filtering is necessary. Without it, the builder would face the raw possibility space and be paralyzed by its scale. The AI's filtering is what makes the interaction productive rather than overwhelming.

But filtering is not neutral. every filter embeds criteria — criteria about what is relevant, what is promising, what is likely to satisfy the requester's intent. These criteria are not transparent to the builder. They are encoded in the model's weights, learned from patterns in the training data, shaped by the architectural decisions of the model's designers. The builder does not see the filter. She sees the output. And the output feels like her decision environment — a natural, unmediated presentation of the alternatives available to her — when in fact it is a designed decision environment, shaped by criteria she did not set and cannot fully inspect.

Simon would have recognized this immediately, because it is the same structure he found in every organization he studied. The municipal administrator who reads reports in the order they arrive does not experience the arrival order as an architectural feature of her decision environment. She experiences it as the way things are — as the natural order of the information available to her. The choice architecture is invisible to the person operating within it. Its influence is exercised not through coercion or persuasion but through the structuring of what appears natural, obvious, and available.

The implications for AI-augmented building are specific and testable. Consider the builder who asks Claude to propose an architecture for a new software system. Claude responds with an approach — perhaps a microservices architecture with specific technology choices, a particular data model, a certain deployment strategy. The builder evaluates the proposal. It is sound. It addresses the requirements she described. It reflects reasonable engineering tradeoffs.

What the builder does not see is the space of alternative architectures that Claude's inference process considered and discarded before producing the response. She does not see the monolithic architecture that might have been simpler and more appropriate for the scale of the project. She does not see the event-driven approach that might have handled the specific concurrency requirements more elegantly. She does not see these alternatives because the AI's filtering has removed them from the decision environment before the builder encountered it.

The removal might be correct. The AI's filtering criteria might accurately identify the best approach for the stated requirements. But the builder cannot verify this, because the verification would require access to the full possibility space that the filter has compressed — access that is unavailable by design. The builder evaluates the proposed architecture against her own knowledge, her own experience, her own pattern library. If her knowledge encompasses the alternatives that Claude filtered out, she may recognize the absence and ask for them explicitly. If her knowledge does not encompass those alternatives — if she is precisely the kind of builder who would benefit most from being exposed to approaches she has not previously considered — then the architecture of the AI's choice environment has narrowed her decision without her awareness.

The dynamic is compounded by a feature of AI output that Simon's organizational research anticipated in a different context: the confidence of the presentation. In Simon's studies of organizational decision-making, he found that the format in which information was presented significantly influenced how it was evaluated. Data presented in a formal report with clear headings, numbered conclusions, and confident language was evaluated differently from the same data presented in a tentative memo with qualifications and uncertainty markers. The format signaled authority. The authority influenced evaluation. Bounded minds use presentation quality as a heuristic for content quality — a shortcut that is often efficient but occasionally catastrophic.

AI output is, by default, presented with confidence. Claude does not preface its code with "I'm not sure this is the best approach, but..." It does not flag the alternatives it considered and discarded. It does not mark the boundaries of its competence with uncertainty indicators proportional to its actual uncertainty. The output arrives polished, well-structured, and rhetorically complete — a format that bounded minds interpret, via the presentation-quality heuristic, as authoritative. The builder's evaluation is shaped not only by what the AI presents but by how it presents it. The confidence of the output becomes part of the choice architecture, nudging the builder toward acceptance and away from the kind of skeptical, adversarial evaluation that would detect the errors concealed beneath the polish.

Simon's prescription for this problem is the same prescription he applied to every organizational decision architecture he studied: make the architecture visible. The administrator who recognizes that the arrival order of reports influences her attention can redesign the routing system to present reports by importance. The builder who recognizes that the AI's confident presentation influences her evaluation can design interaction protocols that counteract the effect — requesting alternative approaches explicitly, asking the AI to identify the weaknesses in its own proposal, building evaluation checkpoints that force the builder to interrogate the output before accepting it.

But making the architecture visible is itself a bounded-rationality problem. It requires meta-cognitive effort — the effort to think about one's own thinking, to recognize the influence of the decision environment on the decision, to step outside the frame and see the frame. This meta-cognitive effort competes for the same attentional resources that the evaluation itself demands. The builder who is deeply engaged in evaluating a complex system architecture is using her full attentional capacity for the evaluation. She has no surplus attention available for the meta-cognitive operation of questioning whether the evaluation itself has been shaped by an architecture she did not design.

The result is that the builders most capable of recognizing the AI's choice architecture are the builders who need the recognition least — the experts whose deep pattern libraries already encompass the alternatives that the AI might have filtered out, and whose evaluative heuristics are robust enough to override the presentation-quality bias. The builders who most need to recognize the architecture — the novices, the generalists, the boundary-crossers who are using AI to work in domains where their pattern libraries are thin — are the builders least equipped to recognize it, because the recognition requires precisely the domain expertise they lack.

This asymmetry maps directly onto a pattern that Simon documented across decades of organizational research. He found that the people most influenced by organizational decision architectures were the people least aware of them — junior employees who accepted the organization's framing of problems as natural rather than designed. Senior employees, whose experience extended beyond the current organization, could see the architecture for what it was and work around it. The expertise gap in organizational decision-making was not primarily a gap in information or computation. It was a gap in architectural awareness — the capacity to see the decision environment as a designed structure rather than as the way things naturally are.

AI replicates this pattern at the level of the individual builder's interaction with the tool. The builder who has worked in software architecture for twenty years brings a pattern library deep enough to recognize when Claude's proposed architecture is missing obvious alternatives. The builder who is new to software architecture — or who is a designer or product manager working in a domain that AI has made newly accessible — takes the AI's proposal as the reasonable starting point it appears to be, unaware that the starting point was selected by a filtering process that may or may not align with the actual requirements of the problem.

The design implication is precise and actionable: AI tools should be designed — and builders should be trained — to make the choice architecture visible. This means surfacing the alternatives that the AI considered and did not present. It means providing uncertainty indicators proportional to the AI's actual confidence. It means building interaction patterns that explicitly solicit alternatives to the AI's initial proposal. It means educating builders about the filtering function that the AI performs and the ways that filtering shapes their decisions.

None of these interventions are technically difficult. All of them are architecturally demanding, because they require the AI system to reveal the structure of its own decision process — a form of transparency that current AI systems do not provide by default and that the competitive dynamics of the AI industry do not reward. A system that presents its output with confidence and without alternatives feels more capable than a system that presents its output with uncertainty and multiple options. The market rewards the appearance of capability. The builder's bounded rationality rewards it too — the confident output is easier to evaluate, faster to accept, less demanding on the scarce attentional resources that deep evaluation requires.

The tension between what the market rewards and what bounded rationality requires is the central design challenge of the AI age. Simon spent his career arguing that the design of decision environments is the most consequential form of design — more consequential than the design of the artifacts those decisions produce, because the decision environment determines which artifacts get produced, which alternatives are considered, which tradeoffs are made. The choice architecture of AI tools is, by this standard, the most important design challenge that the technology industry has ever faced. The answers embedded in the architecture — which alternatives to present, which to filter, how to signal uncertainty, how to structure the interaction between bounded mind and unbounded tool — will shape the quality of every decision made within that architecture for as long as the tools are used.

Philip Tetlock

"The best forecasters see themselves not as seers but as calibrators — constantly updating their beliefs to match the evidence rather than the evidence to match their beliefs."

Superforecasting · 2015

→ Philip Tetlock — On AI: Chapter 4

Simon believed that design could be studied rigorously, that the science of the artificial deserved the same respect as the science of the natural, and that the gap between how things are and how they should be was a gap that rigorous thinking could narrow. The choice architecture of AI is the test case for that belief. The architecture exists. It shapes decisions at scale. It is largely unexamined, operating below the threshold of awareness of the builders who work within it.

Making it visible, making it legible, making it a subject of deliberate design rather than an accidental byproduct of training — this is the Simonian project for the AI age. The bounds of rationality remain. The architecture that channels those bounds toward wisdom or toward waste is the variable that human designers still control. Whether they exercise that control, or cede it to the default dynamics of a system designed without attention to how bounded minds will interact with it, will determine the quality of what gets built in the decade ahead.

Related You On AI Encyclopedia Topics for This Chapter

16 related entries — click to explore the full topic catalog

+

Every one of the 16 Orange Pill Wiki entries this chapter links to — the people, ideas, works, and events it uses as stepping stones. Click any card for the full entry.

Concept (16)

Concept

Administrative Behavior

Concept

Cognitive Ease

Concept

Cognitive Restructuring by AI

Concept

Default Settings as Cognitive Governance

Concept

Deliberate Practice

Concept

Designing for the Bounded Builder

Concept

Equivocality

Concept

Inclusive Institutional Design

Concept

Loss Aversion

Concept

Metacognition — Thinking About Thinking

Concept

Path Dependence

Concept

Technical Debt

Concept

The Algorithmic Cocoon as Unfreedom

Concept

The Availability Heuristic

Concept

The Extended Mind

Concept

The Framing Effect

Chapter 5: The Science of the Artificial

In 1969, Herbert Simon published a slim, ambitious volume that proposed something no one in the academy had thought to propose: that designed things deserve their own science. Not the science of physics, which studies the world as it is. Not the science of biology, which studies organisms shaped by natural selection over millions of years. A different science entirely — the science of things that exist because someone decided they should, things shaped not by natural law but by human purpose, things that operate at the interface between what their designers intended and what the world demands.

Simon called it the science of the artificial. The word "artificial" was chosen with care. Not synthetic, which implies imitation. Not technological, which implies machinery. Artificial — made by art, by human design, for human ends. A bridge is artificial. A corporation is artificial. A computer program is artificial. A legal code, a curriculum, an economic policy — all artificial in Simon's precise sense: systems designed to achieve purposes in environments that the designer did not create and cannot fully control.

The conceptual architecture of The Sciences of the Artificial rests on a single distinction that has become, in the age of AI, the most important distinction in the philosophy of technology. Simon distinguished between an artificial system's inner environment — its design, its architecture, its internal logic — and its outer environment — the world in which it operates, the problems it addresses, the humans it serves, the constraints it faces. The behavior of the system, Simon argued, reflects not the inner environment alone and not the outer environment alone but the interface between them. The system succeeds when its inner environment is well-adapted to the demands of its outer environment. It fails when the two diverge — when the internal logic of the design does not match the external requirements of the world.

A bridge whose inner environment — its structural calculations, its material properties, its load-bearing architecture — matches the outer environment — the span it must cross, the traffic it must bear, the weather it must withstand — stands. A bridge whose inner environment diverges from its outer environment — designed for a lighter load than it must carry, built from materials unsuited to the climate — collapses. The success or failure of the artifact is determined at the interface. Not inside the system. Not outside it. Between them.

The framework is general enough to apply to any designed system, but Simon developed it with a specific class of artifacts in mind: information-processing systems, both human and computational. A human organization is an artificial system whose inner environment — its reporting structure, its decision rules, its allocation of authority — must interface with the outer environment of the market, the regulatory landscape, the competitive terrain. A computer program is an artificial system whose inner environment — its algorithms, its data structures, its computational logic — must interface with the outer environment of the problem it addresses, the users it serves, the hardware it runs on.

The builder's task, in Simon's framework, is not to build the inner environment in isolation. It is to manage the interface — to ensure that the designed logic of the artifact remains adapted to the demands of the world outside it. This management is what Simon meant by design, and he insisted that design, properly understood, is a form of knowledge as rigorous as any natural science. The engineer who designs a bridge is not merely applying physics. She is making choices about the interface between physics and the world — choices that require judgment, experience, and the kind of integrative thinking that no formula can replace.

AI is an artificial system in Simon's precise sense, and its interface dynamics are more complex than any artifact Simon studied during his lifetime.

The inner environment of a large language model is its training data, its architecture, its weights — the billions of parameters adjusted during training to produce outputs that match the patterns in the data. This inner environment is not designed in the traditional sense. It is not specified by an engineer who decides what each parameter should be. It is learned — shaped by a training process that optimizes for pattern-matching across a vast corpus of text. The designer specifies the architecture and the training procedure. The inner environment that results is an emergent property of the interaction between architecture, procedure, and data.

This means that the inner environment of an AI system is opaque in a way that Simon's bridges and organizations were not. The engineer who designs a bridge can inspect every structural element and trace the logic from load to beam to foundation. The administrator who designs an organizational hierarchy can examine every reporting relationship and predict how information will flow. The designer of an AI system cannot inspect the billions of parameters that constitute the model's inner environment and trace the logic from input to output. The inner environment is too complex, too distributed, too far from human-legible representation for the kind of inspection that traditional design permits.

The outer environment is more tractable but no less complex. It consists of every problem the builder brings to the AI, every user who will interact with the products the AI helps build, every organizational constraint that shapes what gets built and how, every ethical consideration that the builder may or may not recognize, every downstream consequence that the builder may or may not foresee. The outer environment is, by definition, everything the inner environment must be adapted to — and it is vast, dynamic, and partially unknowable.

The builder operates at the interface. She brings a problem from the outer environment — a feature to build, a system to design, an argument to construct — and presents it to the inner environment of the AI. The AI processes the input through its opaque internal logic and produces an output. The builder evaluates the output against the demands of the outer environment — the users' needs, the system's constraints, the quality standards, the ethical requirements — and accepts, modifies, or rejects it.

This evaluation is the critical operation. It is where the interface is managed. It is where the builder determines whether the inner environment of the AI has produced an output adapted to the outer environment of the world. And it is where bounded rationality constrains the process most severely, because the evaluation requires the builder to hold in mind simultaneously the demands of the outer environment (which are complex, context-dependent, and partially tacit) and the properties of the inner environment's output (which are visible in the output itself but whose relationship to the inner logic that produced them is opaque).

The evaluation, in other words, requires the builder to assess an artifact whose inner logic she cannot fully inspect against requirements that she cannot fully articulate. Both sides of the interface are partially opaque. The builder's judgment must bridge the gap.

Simon would have recognized this as a familiar structure. He encountered it in every organization he studied. The executive who evaluates a strategic recommendation must assess an artifact — the recommendation — whose inner logic she cannot fully inspect (the analysis was performed by a team whose methods and assumptions she has only partially reviewed) against requirements she cannot fully articulate (the strategic landscape is too complex, too dynamic, too dependent on unknowable future conditions for any complete specification of what "the right strategy" would look like). The evaluation operates under double opacity: opacity of the inner environment and opacity of the outer environment. The executive's judgment bridges the gap.

The difference, in the AI age, is the scale of the opacity. The strategic recommendation was produced by a team of bounded minds whose reasoning, while not fully inspectable, could in principle be interrogated through conversation, through review of the analysis, through the organizational mechanisms that Simon spent his career studying. The AI's output was produced by a process that cannot, even in principle, be fully interrogated by the builder. The inner environment of a large language model is not merely complex. It is complex in a way that resists the kind of inspection that human organizations, however imperfectly, permit.

This means the builder's interface-management task is harder than any previous design task in the history of artificial systems. She must evaluate outputs whose provenance she cannot trace, generated by logic she cannot inspect, against requirements that are themselves partially tacit. The only resource she has for this evaluation is her own judgment — the pattern libraries, the domain expertise, the evaluative heuristics that constitute her professional capability. And these resources are bounded.

The science of the artificial, as Simon conceived it, is the rigorous study of how this interface should be managed. It asks: given the inner environment of the AI (opaque, powerful, pattern-based) and the outer environment of the world (complex, dynamic, partially unknowable), how should the builder's interaction with the AI be designed to maximize the probability that the outputs serve the world's actual needs?

The question is not answered by making the AI more powerful. A more powerful inner environment does not simplify the interface-management task if the outer environment's demands remain complex and the builder's evaluative resources remain bounded. A more powerful AI produces outputs that are, on average, better adapted to the outer environment — but "on average" conceals the variance that matters. The cases where the AI's output is poorly adapted — where the inner logic has produced something that satisfies the patterns in the training data but fails the demands of the specific situation — are precisely the cases where the builder's evaluative judgment is most critical and most strained.

The question is also not answered by making the builder more knowledgeable, though deeper expertise helps. The fundamental asymmetry of the interface — opaque inner environment, complex outer environment, bounded evaluator — persists regardless of the builder's skill level. The expert builder manages the interface better than the novice builder, because her pattern libraries are deeper and her evaluative heuristics are more refined. But she still operates under bounded rationality. She still cannot inspect the AI's internal logic. She still cannot fully articulate the outer environment's requirements. The interface remains partially opaque on both sides, and judgment remains the only bridge.

Simon's answer — the answer that constitutes the core of the science of the artificial — is that the interface itself must be designed. Not just the inner environment (the AI) and not just the outer environment (the world), but the interaction structure through which the bounded builder mediates between them. This means designing the conversation between builder and AI as a decision architecture. It means structuring the interaction so that the builder's bounded attention is directed toward the evaluation points that matter most — the places where the AI's output is most likely to diverge from the outer environment's demands, the decisions where the tradeoffs are most consequential, the junctures where the builder's judgment adds the most value.

This is what You On AI describes as the "creative director" role — the builder whose value lies not in generating the output but in directing the process that generates it, evaluating the output against the world's demands, and making the judgment calls that determine whether the artifact serves its purpose. Simon's framework provides the theoretical infrastructure for this role: the creative director is the interface manager between the AI's inner environment and the world's outer environment. Her value is precisely her capacity to bridge the double opacity — to hold in mind simultaneously what the AI has produced and what the world requires, and to judge whether the two are adequately matched.

Peter Drucker

"The most important thing in communication is hearing what isn't said."

Various attributed sources ·

→ Peter Drucker — On AI: Chapter 5

The science of the artificial, applied to AI, yields a research agenda that is barely in its infancy. How should the interaction between builder and AI be structured to maximize evaluative quality? What evaluation protocols — what sequences of questions, what checkpoints, what adversarial tests — produce the most reliable assessments of the AI's output against the world's requirements? How should the AI's filtering function (the choice architecture of the previous chapter) be designed to present alternatives in ways that respect the builder's bounded attention while exposing the dimensions of the decision that require the builder's judgment? How should organizations be designed to distribute the interface-management task across multiple bounded minds, recovering through collective intelligence what individual bounded rationality cannot provide?

These are design questions, not natural-science questions. They are questions about how things should be built, not about how things are. Simon spent his career arguing that such questions deserve the same rigor, the same respect, the same institutional investment as the questions of physics and biology. The argument has never been more pressing. The artifact under design — the interaction between human intelligence and artificial intelligence — is the most consequential artifact in the history of designed systems, because every other artifact of the coming decades will be produced through it. The quality of the meta-design — the design of the design process — determines the quality of everything downstream.

Simon began his 1969 book with an observation about the peculiar status of design in the modern university: it had been all but eliminated from the curriculum, displaced by the natural sciences that could claim a rigor that design, as traditionally taught, could not match. Engineering schools taught physics. Business schools taught economics. Computer science departments taught mathematics. The art of design — the integrative, judgment-intensive, interface-managing art that Simon considered the distinctive competence of the professional — had been squeezed out by disciplines that studied the world as it is rather than the world as it might be.

The observation was prescient. The generation of builders now entering the AI age has been trained in the natural sciences of computation — algorithms, data structures, machine learning theory — but not in the science of the artificial that Simon spent his career advocating: the science of how designed systems should be structured to serve human purposes in complex environments. The gap between what builders know how to build and what they know about how to evaluate whether what they have built actually serves the world it was built for is wider than it has ever been, because the building has been accelerated by AI while the evaluating remains as bounded — and as undertaught — as it was before the machines arrived.

Closing that gap is the project of a generation. Simon provided the framework. The AI age provides the urgency. Whether the institutions respond — whether universities teach the science of design with the seriousness they currently reserve for the science of computation, whether organizations invest in evaluation architecture with the resources they currently invest in generation capability — remains the open question. Simon would have noted, with the dry precision that characterized his intellectual style, that the question is itself a design problem, and that design problems, unlike natural-science problems, do not solve themselves. They require the intervention of bounded minds making choices about how things should be, and accepting responsibility for the consequences of those choices in a world more complex than any mind can fully comprehend.

Related You On AI Encyclopedia Topics for This Chapter

14 related entries — click to explore the full topic catalog

+

Every one of the 14 Orange Pill Wiki entries this chapter links to — the people, ideas, works, and events it uses as stepping stones. Click any card for the full entry.

Concept (14)

Concept

Ashby's Law of Requisite Variety

Concept

Boundary Object

Concept

Boundary Work

Concept

Computational Theory of Mind

Concept

Emergence

Concept

Muddling Through

Concept

Near-Decomposability

Concept

Self-Organization

Concept

Systems Thinking

Concept

The Architecture of Complexity

Concept

The Creative Director in the AI Economy

Concept

The Learning Organization

Concept

The Sciences of the Artificial

Concept

Zone of Proximal Development (Vygotsky)

Chapter 6: Near-Decomposability and the Modular Builder

In 1962, Herbert Simon published a paper called "The Architecture of Complexity" that contained, among other contributions, a parable about two watchmakers. The parable has become one of the most cited in the literature on complex systems, not because it is subtle — it is not — but because the principle it illustrates is so fundamental to the organization of complexity that it recurs at every scale of biological, social, and technological systems.

The two watchmakers, Hora and Tempus, each assemble watches containing one thousand parts. They work in environments subject to interruption — a phone rings, a customer arrives, a component slips. Each interruption forces the watchmaker to set down his partially assembled work, and the work falls apart. The difference between the two watchmakers is architectural. Tempus assembles his watches as a single sequence of one thousand steps. Each interruption destroys all prior progress and forces him to start over. Hora assembles his watches in a hierarchy of subassemblies: first, ten components into a stable sub-unit; then, ten sub-units into a larger stable module; then, ten modules into the completed watch. Each interruption destroys only the current sub-assembly — the nine or fewer steps since the last stable unit was completed. The accumulated progress in completed sub-units and modules is preserved.

The arithmetic is devastating. Tempus, working under the same interruption rate, completes watches at a fraction of Hora's rate — not because he is less skilled, not because his components are inferior, but because his architecture is wrong. Hora's hierarchical assembly structure is resilient to interruption. Tempus's sequential structure is catastrophically fragile.

The principle that the parable illustrates is near-decomposability: the property of complex systems in which interactions within subsystems are substantially stronger than interactions between subsystems. The watch is nearly decomposable: the components within a sub-unit interact strongly with each other (they must fit together precisely) but weakly with components in other sub-units (they are connected through well-defined interfaces). This structure makes the system buildable by bounded minds in interruptible environments. Each subsystem can be understood, assembled, and tested largely independently of the others. The builder does not need to hold the entire thousand-component system in mind simultaneously. She needs to hold only the current ten-component sub-unit — a cognitive load compatible with the bounds of human working memory.

Simon observed that nearly decomposable structures are ubiquitous in both natural and artificial systems. Biological organisms are nearly decomposable: organs interact strongly within their own systems (cardiovascular, nervous, digestive) and weakly across systems (the cardiovascular system connects to the nervous system through well-defined interfaces, not through pervasive intermingling). Social organizations are nearly decomposable: departments interact strongly within their boundaries (the engineering team coordinates intensively within itself) and weakly across boundaries (engineering connects to marketing through defined processes — spec handoffs, review meetings, shared dashboards). Computer software is nearly decomposable: modules encapsulate functionality and interact through APIs, not through shared global state.

The ubiquity is not coincidental. Simon argued that near-decomposability is the architecture that bounded rationality requires. A system that is not nearly decomposable — a system in which every component interacts strongly with every other component — cannot be understood by a bounded mind, because understanding any single component requires understanding all of them simultaneously. The cognitive load scales with the total system complexity, which quickly exceeds the capacity of any biological processor. Nearly decomposable systems are the only systems that bounded minds can build, understand, maintain, and modify — because the modular structure allows each mind to operate within a cognitively manageable subsystem while trusting that the interfaces between subsystems will preserve the integrity of the whole.

This is why organizations have the structures they have. The departmental hierarchy, the division of labor, the specialist role — these are not arbitrary impositions of bureaucratic power, though they can serve that function too. They are cognitive architectures. They decompose the organization's total decision-making burden into subsystem-sized chunks that bounded minds can handle. The marketing department holds the marketing problem. The engineering department holds the engineering problem. The interface between them — the product specification, the cross-functional meeting, the shared roadmap — is narrow by design, because narrow interfaces are what near-decomposability requires. Widen the interfaces, and the cognitive load on each participant increases. Make every component of every department's work visible to every other department, and the information burden exceeds any individual's attentional capacity. The modular structure is not a convenience. It is a necessity imposed by the bounds of human cognition.

AI disrupts this architecture by expanding the range of what a single bounded mind can hold.

The expansion is real and consequential. The backend engineer who works with Claude Code can now build frontend interfaces — not because she has learned frontend development in the traditional sense, but because the AI handles the implementation details of the frontend domain while she provides the judgment about what the interface should accomplish. The designer who works with AI can now write functional features — not because he has acquired the pattern libraries of a software engineer, but because the AI translates his design intention into working code. The product manager can now prototype — not because she has become technical, but because the AI converts her product vision into functional artifacts that she can evaluate directly, without the sequential handoff through engineering that the old architecture required.

Each of these capabilities represents a weakening of the boundaries between subsystems. The backend engineer's subsystem and the frontend engineer's subsystem, previously connected through a narrow interface (the API specification, the design document, the cross-functional review), are now connected through a much wider channel — the single mind of the backend engineer, augmented by AI, operating across both domains simultaneously. The nearly decomposable structure has become less decomposable. The interactions between subsystems have intensified. The modular boundaries have blurred.

You On AI documents this blurring extensively. Segal describes engineers reaching across domains, designers writing features, the emergence of "vector pods" that integrate functions previously allocated to separate teams. The organizational chart has not changed, but the actual flow of contribution has shifted beneath it, "like water finding new channels under a frozen surface." The diagnosis is accurate. The near-decomposable structure of the traditional technology organization is dissolving as AI enables individuals to operate across the boundaries that previously defined their cognitive domains.

Simon's framework reveals both the gain and the risk of this dissolution.

The gain is integration. When the modular boundaries weaken, the losses that occur at the interfaces between modules are reduced. Every handoff between teams introduces translation loss — the gap between what the designer intended and what the specification captures, the gap between what the specification describes and what the engineer builds, the gap between what the engineer builds and what the user experiences. Each gap represents a failure of the interface between subsystems. The wider the interface, the more information it must carry, and the more information it carries, the more can be lost in translation. Narrow interfaces preserve cognitive manageability at the cost of translation fidelity. The nearly decomposable structure trades integration for comprehensibility.

When AI enables a single mind to operate across module boundaries, the handoff losses diminish. The designer who builds the feature herself does not lose fidelity at the specification interface, because there is no specification interface — the design intention flows directly into the implementation. The product manager who prototypes her own concept does not wait for the engineering team's interpretation of her vision, because she can evaluate the concept directly in functional form. The integration gain is real, and Segal's account of the Napster Station development — thirty days from concept to functioning product — is evidence of what integration gains can produce when the modular boundaries are removed.

The risk is cognitive overload. The modular boundaries that are dissolving served a cognitive function that the dissolution does not replace. They limited the scope of what any single mind needed to hold. The backend engineer who operated within her subsystem was not burdened with frontend considerations, design judgments, product strategy, or user research. The boundaries insulated her — allowed her bounded attention to focus on the domain where her pattern libraries were deepest and her evaluative heuristics most reliable.

When the boundaries dissolve, the insulation disappears. The backend engineer building frontend interfaces now bears the cognitive load of both domains. She must evaluate not just whether the backend logic is sound but whether the interface serves the user, whether the design is coherent, whether the product strategy is advanced by this feature rather than another. Each additional domain consumes attention. Each unfamiliar domain is evaluated with thinner heuristics, shallower pattern libraries, less reliable judgment.

The senior engineer in You On AI who reported decreased architectural confidence despite increased productivity was experiencing precisely this effect. The dissolution of his modular boundary — his insulation from domains outside his expertise — exposed him to decision domains where his evaluative resources were thinnest. His bounded attention, previously concentrated on the domain where his judgment was most reliable, was now spread across a wider surface. The output increased. The confidence in the output decreased. The tradeoff is characteristic of reduced near-decomposability: more integration, less depth of evaluation at any single point.

Simon's framework predicts a specific response to this dynamic, and the prediction maps onto what organizations are beginning to discover empirically. When near-decomposability decreases — when the modular boundaries weaken — the system must either develop new boundaries at a different level of aggregation or accept the cognitive costs of reduced modularity.

The new boundaries are emerging, though they are not always recognized as such. The "vector pods" that Segal describes — small groups whose job is to decide what should be built rather than to build it — are a new modular boundary. They decompose the organization's total decision burden not by functional domain (engineering, design, marketing) but by decision type (what to build vs. how to build it). The "what" decision and the "how" decision are different cognitive operations, requiring different evaluative resources and different pattern libraries. Separating them into distinct subsystems — with the vector pod handling the "what" and the AI-augmented builder handling the "how" — restores near-decomposability at a higher level of abstraction.

This is the ascending friction principle that You On AI describes, translated into Simon's architectural vocabulary. The modular boundaries have not disappeared. They have moved upward — from the level of implementation domains (backend, frontend, design) to the level of cognitive operations (evaluation, generation, integration). The old modularity decomposed work by what kind of thing you build. The new modularity decomposes work by what kind of thinking you do. The builder who generates, augmented by AI, occupies one module. The evaluator who judges, armed with domain expertise and strategic context, occupies another. The interface between them — the specification of intent, the evaluation of output — is the new narrow channel through which the nearly decomposable system maintains its cognitive manageability.

Whether this new decomposition is stable depends on whether the interfaces between the new modules can carry the information that the evaluation requires. The old interfaces — spec documents, design reviews, cross-functional meetings — were imperfect but legible. The builder could read a specification and know what was expected. The evaluator could review a build and assess whether the specification had been met. The new interfaces are less well-defined. How does a vector pod communicate its strategic intent to an AI-augmented builder? How does the builder communicate the nuances of the AI's output back to the evaluator? These interfaces are being invented in real time, without the benefit of decades of organizational learning that shaped the old ones.

Donella Meadows

"You think that because you understand one you must understand two, because one and one make two. But you must also understand and."

Thinking in Systems · 2008

→ Donella Meadows — On AI: Chapter 6

Simon would have noted that this is exactly the situation his science of the artificial was designed to address. The interfaces between modules in a nearly decomposable system are design problems — problems about how the system should be structured to serve the cognitive needs of the agents operating within it. They are not solved by technology alone. They are solved by the deliberate design of interaction structures, communication protocols, and evaluation practices that respect the bounds of the minds that must use them.

The builder working across dissolved modular boundaries needs new cognitive scaffolding. Not the old scaffolding of departmental structure and sequential handoffs — that scaffolding was designed for the old decomposition and is becoming overhead rather than support. New scaffolding: evaluation checklists that remind the boundary-crossing builder to assess each domain with appropriate rigor; interaction protocols that surface the AI's filtering decisions and expose the alternatives it considered; team structures that pair broad-but-thin generalists with deep-but-narrow specialists, recovering through collaboration the evaluative depth that the individual's boundary-crossing sacrifices.

Hora's watches were buildable because his hierarchical structure matched the interruption environment. Tempus's watches were unbuildable because his sequential structure did not. The lesson is not that hierarchy is always superior to sequence. The lesson is that the architecture must match the cognitive environment. When the environment changes — when AI expands the range of what a single mind can hold while leaving the depth of evaluation bounded — the architecture must change with it. The organizations that recognize this and redesign their modular structure accordingly will build watches. The organizations that maintain the old structure in a new cognitive environment, or that abandon structure entirely in the name of agility, will find that their thousand-component assemblies fall apart at the first interruption.

Related You On AI Encyclopedia Topics for This Chapter

14 related entries — click to explore the full topic catalog

+

Every one of the 14 Orange Pill Wiki entries this chapter links to — the people, ideas, works, and events it uses as stepping stones. Click any card for the full entry.

Concept (14)

Concept

Ascending Friction

Concept

Boundary Object

Concept

Boundary Work

Concept

Creative Destruction

Concept

Distributed Cognition

Concept

Division of Labour

Concept

Getting Things Done (GTD)

Concept

Human Problem Solving

Concept

Jobs to Be Done

Concept

Near-Decomposability

Concept

Problem Space

Concept

Scaffolding

Concept

The Chess Complementarity Shift

Concept

The Extended Mind

Chapter 7: Problem-Solving and the AI Partner

Herbert Simon spent the better part of two decades studying how human beings solve problems. The research, conducted primarily with Allen Newell at Carnegie Mellon from the late 1950s through the 1970s, produced both a theory of human problem-solving and the first computer programs capable of replicating aspects of it. The theory and the programs were deliberately intertwined — Simon and Newell built programs that solved problems in order to understand how humans solve problems, and they studied human problem-solving in order to build better programs. The recursion was not incidental. It was the method.

The theory they developed models problem-solving as search through a problem space. The problem space is a formal representation of the problem: an initial state (where the solver starts), a goal state (where the solver wants to arrive), a set of operators (the legal moves that transform one state into another), and path constraints (the conditions that any valid solution must satisfy). The solver navigates the problem space by selecting operators, applying them to the current state, evaluating the result, and selecting the next operator. The process continues until a state is reached that satisfies the goal criteria — or until the solver gives up.

The critical insight is that the problem space for any non-trivial problem is too large to search exhaustively. A chess game has more possible positions than atoms in the observable universe. A software architecture problem has more possible configurations than any mind — bounded or unbounded — could enumerate. The solver cannot evaluate every path. She must choose which paths to explore and which to abandon, and the quality of those choices determines whether she reaches the goal efficiently, inefficiently, or not at all.

The mechanism that guides these choices is what Simon and Newell called heuristic search. A heuristic is a rule of thumb — a strategy for selecting promising paths and avoiding unpromising ones without the guarantee of optimality that an exhaustive search would provide. The chess grandmaster does not evaluate every legal move. She evaluates four or five, selected by heuristics built from thousands of hours of pattern recognition: "Control the center." "Develop your pieces early." "When ahead in material, simplify; when behind, complicate." These heuristics do not guarantee the best move. They direct the search toward regions of the problem space where good moves are most likely to be found.

The quality of the heuristics determines the quality of the problem-solving. A strong player's heuristics are better — more reliable, more refined, more accurately tuned to the specific structure of chess — than a weak player's heuristics. The difference between a grandmaster and a novice is not that the grandmaster thinks more moves ahead. Research by Simon and William Chase demonstrated that grandmasters and novices look roughly the same number of moves deep. The difference is that the grandmaster's heuristics select better starting points for the search — moves that are more likely to lead to strong positions — because her pattern library, built from years of deliberate practice, is vastly richer.

This framework applies directly to the AI-augmented builder, but it requires careful parsing to understand what AI changes and what it does not.

AI transforms problem-solving by dramatically expanding the problem space that a single builder can explore. The builder who works with Claude Code can attempt problems that no individual could have attempted before — systems that span multiple technical domains, products that integrate capabilities previously distributed across teams, solutions that draw on patterns from fields the builder has never studied. The expansion is real. It follows from the same mechanism that drives every capability gain You On AI documents: the collapse of the implementation barrier that previously gated access to the problem space. When implementation is cheap, the builder can afford to explore regions of the problem space that were previously too expensive to reach.

Simultaneously, AI provides implementation heuristics of extraordinary power. The builder does not need to know how to implement a solution in order to explore whether it is viable. She describes the goal. The AI provides the path. The heuristic that previously required years of domain-specific training — "To build a real-time audio system, use this architecture; to build a user interface with these interaction patterns, use this framework" — is now available to anyone who can describe what they want. The implementation heuristics have been democratized.

The combination of expanded problem space and democratized implementation heuristics produces the sensation that You On AI describes as the collapse of the imagination-to-artifact ratio. The builder can imagine something and see it realized in hours. The search through the problem space, which previously terminated at the boundary of what the builder could implement, now extends to the boundary of what the builder can conceive.

But here is where Simon's framework identifies the constraint that the exhilaration of expanded capability tends to obscure. The implementation heuristics have been transformed. The goal-setting heuristics have not.

In Simon and Newell's problem-solving framework, the solver's task has two components: defining the goal and searching for a path to it. Both components require heuristics. The search heuristics guide the navigation through the problem space toward the goal. The goal heuristics guide the specification of what the goal is — what counts as a solution, what quality standards it must meet, what constraints it must satisfy, what tradeoffs are acceptable.

Implementation heuristics are search heuristics. They guide the builder through the problem space of implementation: how to build the thing, what architecture to use, what patterns to follow. AI provides these with remarkable effectiveness. The builder describes a destination, and the AI provides a route.

Goal heuristics are different. They operate before the search begins. They determine what destination is worth reaching. They answer the question that You On AI identifies as the question of the AI age: not "How do I build this?" but "What should I build?" and, more precisely, "What should I build for whom, to what standard, with what tradeoffs, and in service of what larger purpose?"

These questions are not implementation questions. They are judgment questions. They require the builder to integrate information from multiple domains — user needs, market conditions, technical feasibility, organizational strategy, ethical implications — and synthesize it into a coherent specification of purpose. The integration is a cognitive operation that bounded rationality constrains severely, because it requires holding multiple considerations in mind simultaneously and assessing their relative weight in the specific context of the specific problem.

AI does not perform this integration for the builder. It can provide information relevant to each consideration — market data, technical constraints, user research findings — but the synthesis of that information into a coherent goal requires the kind of judgment that is built from experience, from having seen goals specified well and poorly, from having lived through the consequences of good and bad goal-setting. The pattern library that informs goal-setting heuristics is built through the same slow, experiential process that builds all expert pattern libraries: years of deliberate practice, each episode depositing a thin layer of judgment that accumulates into the intuitive sense of what constitutes a goal worth pursuing.

The asymmetry between implementation and goal-setting illuminates something that the productivity metrics of the AI age tend to obscure. The twenty-fold productivity multiplier that Segal reports from Trivandrum is a measure of implementation throughput — the rate at which goals, once specified, are converted into working artifacts. It does not measure the quality of the goals. The same multiplier applied to a well-specified goal and a poorly specified goal produces, respectively, a excellent product built twenty times faster and a mediocre product built twenty times faster. The multiplier is agnostic. It amplifies whatever goal the builder has specified.

This is the problem-solving version of the amplifier argument that runs through You On AI: AI amplifies the signal, and the signal includes the goal. If the goal is well-specified — if the builder has exercised the judgment necessary to define what success looks like, what tradeoffs are acceptable, what quality standards must be met — then the AI's implementation heuristics carry the project efficiently toward a destination worth reaching. If the goal is poorly specified — if the builder has accepted the first plausible objective without the evaluative rigor that good goal-setting requires — then the AI carries the project efficiently toward a destination that may not be worth reaching at all.

The problem-solving research that Simon and Newell conducted revealed a pattern that has direct implications for how AI-augmented builders should work. They found that expert problem-solvers spend more time on problem representation — the initial phase where the problem is formulated, the goal is defined, and the problem space is structured — and less time on search than novice problem-solvers. The novice dives into the search immediately, eager to start generating solutions. The expert lingers at the representation stage, asking what the problem actually is, whether the goal as initially stated captures what matters, whether the problem space has been structured in a way that makes good solutions findable.

The expert's investment in representation pays off during search, because a well-represented problem directs the search heuristics toward promising regions of the problem space. A poorly represented problem — one where the goal is vague, the constraints are unclear, the structure of the problem space does not match the structure of the real-world situation — produces search that is energetic but misdirected. The solver generates solutions rapidly, but the solutions do not serve the purpose, because the purpose was never adequately defined.

AI exacerbates this dynamic by making the search phase so fast and so cheap that the investment in representation feels unnecessary. When the builder can generate a working prototype in minutes, the discipline of spending hours defining the problem — asking what the real need is, what the constraints are, what quality standard the solution must meet, what downstream consequences the solution might produce — feels like overhead. The machine is ready. The implementation is instant. The temptation to specify the goal loosely and iterate rapidly is nearly irresistible.

But rapid iteration without clear goal specification is not problem-solving. It is problem-evasion — the substitution of generative speed for evaluative depth, the production of many artifacts without the judgment to assess whether any of them solve the problem that actually matters. Simon and Newell documented this pattern in novice problem-solvers forty years before AI existed: the novice who starts generating immediately produces more attempts and worse outcomes than the expert who spends the first thirty minutes understanding the problem.

The AI-augmented builder is, in this specific sense, at risk of becoming a permanent novice: someone whose generative power is expert-level but whose goal-specification remains novice-level, because the discipline of representation — the slow, unglamorous work of defining what you are actually trying to achieve — has been crowded out by the seductive speed of generation.

The prescription follows directly from the problem-solving research. Build representation disciplines into the AI-augmented workflow. Before the first prompt, before the first line of generated code, invest in the operations that the expert problem-solver performs at the representation stage: define the goal precisely, identify the constraints explicitly, specify the quality standards that the solution must meet, anticipate the tradeoffs that the search will encounter.

Karl Weick

"How can I know what I think until I see what I say?"

Sensemaking in Organizations · 1995

→ Karl Weick — On AI: Chapter 7

This investment is not overhead. It is the operation that determines whether the subsequent search — however fast, however powerful, however augmented by AI — arrives at a destination worth reaching. The problem space is vast. The AI provides powerful heuristics for navigating it. But navigation without a destination is motion without progress, and the destination is the one thing the AI cannot specify. It is the builder's irreducible contribution — the goal that the search serves, the purpose that the implementation realizes, the judgment that determines whether the artifact, once built, deserves to exist.

Simon and Newell titled their most ambitious work Human Problem Solving. The emphasis on "human" was deliberate. The problem-solving they studied was not abstract computation. It was the specific, bounded, heuristic-guided search that biological minds perform when they encounter situations that matter to them — situations with stakes, with consequences, with the possibility of failure. AI has transformed the search component of this process beyond recognition. The human component — the goal-setting, the representation, the judgment about what constitutes a problem worth solving — remains as bounded, as consequential, and as irreducibly human as it was when Simon first sat down to study how a municipal administrator decides which report to read next.

Related You On AI Encyclopedia Topics for This Chapter

11 related entries — click to explore the full topic catalog

+

Every one of the 11 Orange Pill Wiki entries this chapter links to — the people, ideas, works, and events it uses as stepping stones. Click any card for the full entry.

Concept (11)

Concept

Algorithm Aversion

Concept

Cognitive Maps

Concept

Creative Tension

Concept

Framing

Concept

Heuristic Search

Concept

Jobs to Be Done

Concept

Mental Models

Concept

Productive Failure

Concept

Sensemaking

Concept

The Capability Trap

Concept

The Competency Trap

Chapter 8: The Ant on the Beach

Among the many images that Herbert Simon introduced into the literature of cognitive science, one has proven more durable and more widely misunderstood than any other. It appears in The Sciences of the Artificial, in a passage about the relationship between the complexity of behavior and the complexity of the environment, and it concerns an ant walking across a beach.

The ant's path, viewed from above, is complex. It twists and turns. It doubles back. It traces an irregular course around obstacles — pebbles, driftwood, ridges of sand carved by the retreating tide. If you recorded the path and presented it to someone without context, the observer might conclude that the ant possesses sophisticated navigational intelligence. The path appears to reflect complex planning, flexible adaptation, perhaps even a kind of strategic reasoning about how to reach a distant goal while avoiding intervening obstacles.

Simon's point was that this conclusion would be wrong. The complexity of the ant's path does not reflect the complexity of the ant. It reflects the complexity of the beach. The ant follows simple rules — proceed toward the goal direction, avoid the immediate obstacle, resume the goal direction after the obstacle is passed. The rules are elementary. The environment is complex. The interaction of simple rules with a complex environment produces behavior that appears sophisticated without being sophisticated in the agent.

The metaphor was designed to make a specific argument about the study of human behavior. Simon contended that much of what appears to be complex cognition in human beings is, similarly, a reflection of the environment's complexity rather than the mind's. A human navigating a social situation, a bureaucratic process, or an unfamiliar city may exhibit behavior that appears to require sophisticated planning and subtle inference. But the behavior may be generated by relatively simple heuristics — rules of thumb, learned patterns, default strategies — applied to a complex environment. The sophistication is in the interaction, not necessarily in the agent.

The argument was controversial when Simon made it, and it remains controversial, because it cuts against the intuitive sense that human behavior, especially intelligent human behavior, is the product of correspondingly sophisticated cognitive processes. The grandmother who navigates a complex family dispute with apparent wisdom may be applying a handful of principles — be fair, listen first, don't take sides — to a complex relational environment. The apparent wisdom of the navigation reflects the complexity of the family, not the complexity of the principles. The principles are simple. The family is complicated. The combination produces behavior that looks like deep insight.

Simon was not arguing that human beings are simple. He was arguing that the correct way to understand behavior is to analyze the interaction between the agent's internal rules and the environment's external structure, rather than attributing all observed complexity to the agent's internal sophistication. The ant on the beach teaches a methodological lesson: before crediting the agent with complex intelligence, examine whether the observed complexity might be a property of the environment the agent is navigating.

The lesson applies to the AI-augmented builder with a directness that is uncomfortable precisely because it is diagnostic.

Consider a builder who describes a software system to Claude Code and receives, in response, a working implementation. The implementation is sophisticated — it handles edge cases, follows architectural best practices, employs design patterns appropriate to the problem domain, produces clean and well-documented code. The builder examines the output, confirms that it works, and deploys it. An observer might credit the builder with the sophistication of the output. The builder herself might experience the output as a reflection of her capability — after all, she specified the system, directed the process, and evaluated the result.

Simon's ant-on-the-beach argument raises the question: where does the sophistication actually reside? The builder's contribution to the interaction was a natural-language description — a specification of what the system should do, perhaps with some constraints on how. The description is important. It defines the destination. But the description is also, relative to the implementation, simple. It is a handful of sentences or paragraphs. The implementation is thousands of lines of code, each one reflecting decisions about architecture, data flow, error handling, and performance that the description did not specify and the builder may not have considered.

The sophistication of the output reflects the complexity of the tool — its training data, its pattern libraries, its capacity to translate a natural-language description into a coherent implementation. The builder's contribution, while essential — without the description, the tool produces nothing — may be more analogous to the ant's simple rules than to the beach's complex terrain. The builder provides direction. The tool provides sophistication. The combination produces output that appears to reflect the builder's expertise when it may primarily reflect the tool's.

This is not an insult to the builder, any more than Simon intended the ant metaphor as an insult to the ant. The ant's rules are well-adapted to its environment. The ant survives, reproduces, and contributes to its colony. The rules work. But understanding what is working — the rules or the environment — matters for assessing what happens when the environment changes, or when the rules are applied to an environment they were not designed for.

The distinction matters because it predicts different outcomes for builders with different levels of evaluative capability. The builder whose contribution is primarily directional — who provides the goal specification and relies on the AI for everything else — is operating with simple rules in a complex tool-environment. Her output will appear sophisticated as long as the tool's pattern libraries are well-matched to the problem domain. When they are not — when the problem requires patterns outside the training data, when the domain has unusual constraints that the tool's heuristics do not accommodate, when the correct solution deviates from the common patterns that the tool has been trained to reproduce — her output will fail, and she may not recognize the failure, because recognizing it would require precisely the domain expertise that she relied on the tool to provide.

The builder whose contribution is evaluative — who provides not just the goal but the judgment to assess whether the AI's implementation actually achieves it, who can recognize when the tool's pattern-matching has produced something that looks correct but is structurally flawed, who brings the domain expertise necessary to catch the errors that confident output conceals — is operating with complex rules in a complex tool-environment. Her output is genuinely sophisticated, not because the implementation is hers but because the evaluation is hers, and the evaluation is the operation that determines whether the output serves its purpose or merely resembles output that serves its purpose.

The difference between these two builders is invisible in the productivity metrics that currently dominate the discourse about AI-augmented work. Both builders produce working implementations at the same speed. Both ship features at the same rate. Both achieve the same twenty-fold productivity multiplier. The metrics measure output volume. They do not measure output quality in the dimension that matters — the dimension of whether the output is genuinely adapted to the problem or merely pattern-matched to a superficial description of it.

Simon's chess research illuminates the mechanism. Chase and Simon demonstrated that chess expertise is not a matter of deeper search but of better recognition. The grandmaster and the novice look roughly the same number of moves ahead. The grandmaster selects better starting points for the search because her pattern library — built from roughly fifty thousand to one hundred thousand chunks of knowledge accumulated over at least ten years of deliberate practice — identifies the most promising regions of the problem space before the conscious search begins. The grandmaster does not think harder. She sees differently. The board position activates stored patterns that generate candidate moves and positional assessments faster than conscious deliberation could produce them.

The AI-augmented builder's pattern library serves the same function in the evaluative role. The experienced software architect who examines Claude's output does not analyze it line by line, at least not initially. She sees it — the architectural patterns activate stored knowledge about what works and what fails in similar systems, generating an assessment that feels like intuition but is pattern-recognition at speed. She recognizes the microservices pattern and knows, from experience, that it introduces coordination complexity that the system's scale may not justify. She spots the data model and knows, from pattern, that it will produce performance problems under the load profile the system is likely to encounter. She identifies the error-handling approach and knows, from hard-won failure experience, that it will silently swallow the class of errors most likely to cause production incidents.

The novice builder sees none of this. Not because the information is hidden — the code is right there, every line visible — but because the pattern library that makes the information legible has not been built. The novice sees working code. The expert sees working code that will fail under conditions the novice has not yet learned to anticipate. The difference is not in the code. It is in the evaluator.

This is the ant-on-the-beach problem applied to evaluation rather than generation. The novice evaluator, applying simple rules — "Does it run? Does it pass the tests? Does it match the specification?" — to the complex output of an AI system, produces evaluation that appears adequate. The output runs. It passes tests. It matches the specification. The evaluation looks thorough. But the adequacy of the evaluation reflects the simplicity of the rules, not the thoroughness of the assessment. The complex output has been evaluated with simple criteria, and the simple criteria have missed the structural weaknesses that only complex evaluative heuristics — built from years of experience — can detect.

The implications for the democratization argument that You On AI advances are significant and nuanced. Segal argues, correctly, that AI democratizes generation — it enables people who were previously excluded from building by lack of technical skill or institutional access to produce working artifacts. The developer in Lagos can now build software that, measured by output quality, competes with software built by engineers at well-resourced companies. The imagination-to-artifact ratio has collapsed for everyone.

But the ant-on-the-beach analysis suggests that what has been democratized is the appearance of sophisticated output, not necessarily the substance of it. The developer in Lagos, like the developer at Google, produces output whose sophistication reflects the tool's pattern libraries more than the builder's evaluative depth. The democratization is real at the generation level. Whether it is real at the evaluation level — whether the expanded access to generative power is accompanied by the evaluative capability necessary to ensure the output actually serves its purpose — depends entirely on the builder's pattern library, which is built through experience and cannot be purchased at any price.

Simon would not have opposed democratization. He spent his career studying how bounded minds can be supported by well-designed architectures, and AI is the most powerful supportive architecture ever built for bounded minds. But he would have insisted on distinguishing between the sophistication that the tool provides and the judgment that the builder provides, and he would have noted, with characteristic precision, that confusing the two produces a specific kind of risk: the risk of confident inadequacy, of output that appears to reflect expertise it does not contain, of systems built by builders who cannot evaluate what they have built because the evaluation requires pattern libraries that the tool provides for generation but not for assessment.

The ant navigates the beach successfully because the beach, for all its complexity, has a structure that the ant's simple rules can exploit. The ant walks forward, turns at obstacles, resumes course. The beach is irregular but traversable. The rules and the environment are matched. The AI-augmented builder navigates the problem space successfully when the problem space's structure matches the patterns in the AI's training data and the builder's evaluative rules are sufficient to catch the cases where it does not. When both conditions hold, the output is genuinely good. When either fails — when the problem departs from the training patterns, or when the builder's evaluative heuristics are too simple to catch the departure — the output looks good without being good, and the appearance is the most dangerous thing about it.

The prescription is not to abandon the beach. The beach is where the work is. The prescription is to know what the ant contributes and what the beach contributes — to understand, with Simonian precision, which sophistication is yours and which belongs to the tool. The builder who knows the difference can direct the tool wisely, evaluate its output critically, and recognize the moments when her simple rules are insufficient for the complex terrain. The builder who does not know the difference navigates confidently until the terrain shifts, and then wonders why the path that looked so promising led somewhere she never intended to go.

Related You On AI Encyclopedia Topics for This Chapter

10 related entries — click to explore the full topic catalog

+

Every one of the 10 Orange Pill Wiki entries this chapter links to — the people, ideas, works, and events it uses as stepping stones. Click any card for the full entry.

Concept (10)

Concept

Cognitive Offloading

Concept

Emergence

Concept

Human in the Loop

Concept

Situated Cognition

Concept

Technical Debt

Concept

The Ant on the Beach

Concept

The Chess Complementarity Shift

Concept

The Judgment Bottleneck

Concept

The Skill Premium

Concept

Zone of Proximal Development (Vygotsky)

Chapter 9: Designing for the Bounded Builder

Herbert Simon spent the final decades of his career arguing, with increasing urgency and decreasing patience, that design is a form of knowledge. Not a lesser form — not a craft to be tolerated alongside the real sciences, not an art too subjective for rigorous analysis — but a form of knowledge with its own methods, its own standards of rigor, and its own domain of application. The domain was vast: every human-made system, every institution, every tool, every organizational structure, every curriculum, every policy. Anything that exists because someone decided it should exist, rather than because natural law compels it, falls within the science of design. And the central question of design, as Simon formulated it, is not "How does this system work?" but "How should this system be structured, given the cognitive characteristics of the agents who will operate within it?"

The question is normative. It is about how things ought to be. And it is constrained by a descriptive reality: the agents who operate within designed systems are boundedly rational. They have limited attention, limited working memory, limited evaluative capacity, and limited time. Any design that ignores these limitations — that assumes the agents within the system will behave as perfectly rational optimizers — will fail, not because the design is technically flawed but because it is incompatible with the minds it was designed for.

Simon watched this incompatibility produce failure after failure across decades of organizational research. information systems that delivered everything the decision-maker might need, rather than filtering for what the decision-maker actually required, and that consequently overwhelmed rather than supported. Organizational structures that assumed managers could integrate information from every department simultaneously, rather than providing the hierarchical filtering that bounded attention requires. Decision support tools that presented every alternative without guidance about which alternatives merited the most careful evaluation, and that consequently produced the paradox of choice: more options, worse decisions.

The pattern was always the same. The system was designed for an idealized agent — one with unbounded attention, unlimited evaluative capacity, perfect ability to integrate information from multiple sources simultaneously. The actual agent was bounded. The mismatch between the design and the agent produced not empowerment but overload, not better decisions but more exhausting ones, not the liberation of capability but the drowning of judgment in information that the judgment did not have the resources to process.

AI tools, as currently designed and deployed, replicate this pattern at a scale Simon would have found both fascinating and alarming. The default interaction between a builder and an AI system is optimized for generative power, not evaluative support. The AI produces its best output. It presents it confidently. It offers more if asked. It does not, by default, structure the interaction to conserve the builder's attention, direct the builder's evaluation toward the decisions that matter most, or impose the kind of reflective pauses that prevent the builder's satisficing threshold from rising past the point of diminishing returns.

The design challenge of the AI age is to reverse this default — to build interaction structures, organizational architectures, educational practices, and personal disciplines that treat the builder's bounded attention as the scarce resource it is, and that allocate that resource with the same deliberateness that a well-designed organization allocates budget or personnel.

The challenge operates at three scales: the organization, the educational institution, and the individual. Each scale requires its own design approach, because the constraints that bounded rationality imposes manifest differently at each level.

At the organizational scale, the design challenge is to rebuild the attention architecture that the old hierarchy provided. The traditional organizational hierarchy was, as the previous chapters have argued, a cascading attention filter — each level of the hierarchy condensing information for the level above it, so that each successive decision-maker faced a manageable, evaluated set of alternatives rather than the raw output of the entire organization. AI has disrupted this architecture by enabling individuals to produce output that bypasses the filtering layers, flowing directly to decision-makers whose attention was previously protected.

The organizational design response is not to restore the old hierarchy — the old hierarchy was designed for the old bounds, and reimposing it would sacrifice the integration gains that AI provides. The response is to build new filtering mechanisms suited to the new cognitive environment. Simon's work on organizational decision-making suggests several principles for this design.

The first is the principle of evaluated alternatives. Instead of presenting decision-makers with raw AI-generated output and expecting them to evaluate it from scratch, organizations should build evaluation layers that pre-process the output. The "vector pods" that You On AI describes represent one version of this — small groups whose function is evaluative rather than generative, who assess the AI-augmented output of individual builders before it reaches the strategic decision-makers. The vector pod is a new kind of filter, adapted to an environment where the volume of competent output has exploded and the evaluative capacity of the people who must decide what to do with it has not.

The second principle is sequential evaluation gates. Simon's research on problem-solving consistently demonstrated that expert decision-makers proceed through problems in stages, completing one phase before beginning the next, because staged processing reduces the cognitive load at each stage. Organizations should impose similar staging on AI-augmented workflows — not as bureaucratic checkpoints that slow the work, but as evaluation structures that ensure each phase of the work has been assessed before the next phase begins. The builder generates a system architecture. The architecture is evaluated against the requirements before implementation begins. The implementation is evaluated against the architecture before deployment. Each gate imposes a focused evaluation — a moment where bounded attention is directed at a specific question — rather than leaving the evaluation to accumulate into a single overwhelming assessment at the end.

The third principle is protected evaluation time. The Berkeley researchers who studied AI-augmented work documented that AI-assisted tasks tend to colonize previously protected cognitive space — lunch breaks, transition periods between meetings, the small gaps in the day that previously served as informal cognitive recovery. The organizational design response is to protect evaluation time explicitly — to schedule blocks of time that are designated for the evaluative work that AI-augmented building requires, and to protect those blocks from the generative work that would otherwise fill them. This is the "AI Practice" framework that the Berkeley researchers recommended, reconceived as an organizational attention-conservation mechanism: structured time in which the builders set the AI aside and engage directly with the work of assessing what they have produced, examining its quality, testing its assumptions, and making the kind of slow, integrative judgments that the speed of generation tends to crowd out.

At the educational scale, the design challenge is to build the cognitive capacities that evaluation requires — the pattern libraries, the meta-heuristic capabilities, the domain expertise that determine whether a bounded mind can assess AI-generated output wisely or only superficially.

Simon's research on expertise acquisition established that expert-level pattern libraries require approximately ten years of deliberate practice to build. This timeline has not changed, and Simon's framework suggests that it cannot be compressed, because the biological substrate on which pattern-recognition knowledge is built has its own temporal requirements. The chunks of knowledge that constitute expert evaluation — the fifty thousand to one hundred thousand patterns that a chess grandmaster or a seasoned software architect has accumulated — are deposited through experience, each episode adding a thin layer that the next episode builds upon. The process is inherently serial. It cannot be parallelized any more than physical growth can be parallelized.

But the composition of what is practiced can change. If the binding constraint has shifted from generation to evaluation, then educational curricula should shift with it. Instead of training students primarily in the mechanics of production — how to write code, how to draft legal briefs, how to construct financial models — curricula should invest proportionally more time in the mechanics of evaluation. How to assess whether code is well-architected, not merely functional. How to judge whether a legal argument is genuinely persuasive or merely rhetorically polished. How to determine whether a financial model's assumptions are sound or merely internally consistent.

This shift requires a different kind of pedagogy. evaluation cannot be taught through lectures or textbooks alone, because evaluative expertise is built through the same experiential process that builds all expert pattern libraries: exposure to cases, practice at assessment, feedback on the quality of the assessment, and gradual refinement of the evaluative heuristics through accumulated experience. The classroom that builds evaluative capability is one where students practice evaluating — where they are presented with AI-generated output and asked not to improve it but to assess it, to identify its strengths and weaknesses, to determine whether it serves the purpose for which it was produced, and to articulate the criteria by which they made their assessment.

The teacher who shifted from grading essays to grading questions — the example from You On AI — represents a move in this direction, reconceived through Simon's evaluative framework. The teacher is not merely testing whether students can ask good questions. She is building evaluative pattern libraries — training students to assess what they do not know, to recognize the boundaries of their understanding, to identify the gaps that the AI's confident output might conceal. These are the meta-heuristic capabilities that the previous chapters identified as the critical cognitive resource of the AI age: the ability to evaluate not just whether an answer is correct but whether the question was the right one, not just whether the output works but whether it works for the right reasons.

At the individual scale, the design challenge is the most intimate and the most demanding, because it requires the builder to design her own cognitive environment — to structure her own attention, her own evaluation practices, her own interaction with the AI — with the same deliberateness that Simon brought to organizational design.

The individual builder who works with AI faces the same design choice that every organization faces: treat the AI as an unbounded generative resource and let the bounded mind cope as best it can, or treat the AI as a component in a deliberately designed cognitive system where the builder's bounded attention is the constraint that the entire system is structured to respect.

The second approach requires specific practices. The builder sets explicit goals before engaging the AI — not vague directions but precise specifications of what she is trying to achieve, what quality standards the output must meet, and what tradeoffs she is willing to accept. She imposes evaluation checkpoints on her own workflow — moments where she stops generating and assesses what has been generated against the criteria she specified. She maintains awareness of the AI's choice architecture — the filtering, the defaults, the confident presentation that shapes her evaluation — and counteracts its influence through deliberate adversarial practices: requesting alternatives, asking the AI to critique its own output, seeking out the weaknesses in the implementation before accepting it.

These practices are not natural. They work against the grain of the interaction's default dynamics, which reward speed, reward volume, reward the seductive pleasure of watching an idea become an artifact in real time. The practices impose friction on a frictionless process, and the friction is the point. The friction is what conserves attention. The friction is what forces the evaluation that the satisficing calculus would otherwise skip. The friction is what ensures that the builder's bounded judgment is actually exercised, rather than overwhelmed by the abundance that the unbounded tool produces.

Simon argued throughout his career that the design of systems for bounded agents is the most important and most neglected form of design. The AI age has made the argument impossible to neglect. The bounded agents are still bounded. The systems they work within are more powerful than any systems in human history. The gap between what the systems can produce and what the agents can wisely evaluate is wider than it has ever been. Closing that gap — through organizational architecture, through educational practice, through individual discipline — is the design challenge that will determine whether AI-augmented work produces a generation of extraordinary builders or a generation of extraordinary output that no one has properly assessed.

The science of the artificial was always, at bottom, a science of humility — a discipline premised on the recognition that the agents within the system are finite, that their finitude constrains what the system can achieve, and that the proper response to this constraint is not to wish it away or to pretend it does not exist but to design around it, with intelligence, with rigor, and with the kind of patient attention to human limitation that only a scientist who genuinely understood both the power and the bounds of the human mind could bring.

Related You On AI Encyclopedia Topics for This Chapter

15 related entries — click to explore the full topic catalog

+

Every one of the 15 Orange Pill Wiki entries this chapter links to — the people, ideas, works, and events it uses as stepping stones. Click any card for the full entry.

Concept (15)

Concept

Ashby's Law of Requisite Variety

Concept

Attentional Infrastructure

Concept

Cognitive Patience

Concept

Default Settings as Cognitive Governance

Concept

Deliberation

Concept

Democratic Deliberation

Concept

Enactment

Concept

Information Asymmetry

Concept

Legitimate Peripheral Participation

Concept

Mental Models

Concept

Path Dependence

Concept

Scaffolding

Concept

Self-Efficacy

Concept

The Framing Effect

Concept

The Sciences of the Artificial

Chapter 10: What Remains Bounded When Everything Else Expands

The ledger has two columns, and filling them out requires the precision that Herbert Simon's framework demands. In the left column: what AI has unbound. In the right column: what remains bounded. The exercise is not academic. The left column determines what the AI age makes possible. The right column determines what makes it valuable.

In the left column: information access. The builder who works with a large language model has access, through conversation, to a compressed representation of a substantial fraction of human knowledge. The information constraint that bounded rationality imposed — the limited data available to any single decision-maker — has been relaxed by several orders of magnitude. Not eliminated. The training data has boundaries, biases, and temporal limits. But the relaxation is large enough to constitute a qualitative change in the information environment that any individual builder inhabits.

In the left column: computational implementation. The builder who describes a system in natural language and receives a working implementation has outsourced the computational labor that previously consumed the majority of the building process. The implementation constraint — the months or years required to convert a design into a working artifact — has been compressed to the duration of a conversation. Not for every problem. Not at production quality in every domain. But for a wide enough range of problems, at sufficient quality, to constitute a structural change in what individuals can accomplish.

In the left column: generative breadth. The builder who works across domains — the backend engineer building frontend interfaces, the designer writing features, the product manager prototyping — has access to the AI's pattern libraries across all those domains simultaneously. The disciplinary boundaries that previously gated access to different regions of the problem space have been weakened. Not removed. The AI's competence varies across domains, and the builder's evaluative capability varies more. But the gating function of specialized training — the years of investment required before a mind could produce competently in a given domain — has been loosened for a significant class of work.

These three expansions — information, implementation, and breadth — constitute the genuine revolution. They are not imaginary. They are not incremental. They represent a structural change in the relationship between human intention and human capability, and every testimony in You On AI confirms their reality. The twenty-fold productivity multiplier. The thirty-day product development. The individual builder shipping what previously required a team. These are consequences of the left column, and they are real.

Now the right column.

Attention remains bounded. This has been the recurring theme of every chapter in this volume, because it is the recurring finding of every study, every framework, and every empirical observation that Simon's theory of bounded rationality generates when applied to the AI age. The builder's capacity to attend — to hold something in consciousness, to evaluate it against criteria, to exercise the focused cognitive engagement that separates assessment from mere perception — has not expanded. Four to seven chunks of working memory. A serial processing architecture that can deeply evaluate one thing at a time. A biological attentional system that fatigues, that is subject to distraction, that requires recovery time proportional to the intensity of its use. AI has not altered any of these parameters, and no foreseeable AI system will, because they are properties of biological consciousness, not of the information environment.

Judgment remains bounded. Judgment, in Simon's framework, is the integration of multiple considerations into a single evaluative act — the capacity to assess whether an alternative satisfies multiple criteria simultaneously, to weigh competing values against each other, to resolve tradeoffs that no formula can specify. AI provides information relevant to each consideration. It does not perform the integration. The integration is a cognitive operation that requires the kind of tacit knowledge — the pattern libraries, the experienced-based intuitions, the domain expertise that manifests as "taste" — that is built through years of practice and cannot be transferred through conversation. The builder's judgment is as bounded as it was before the AI arrived. It is merely more consequential, because the volume of output that the judgment must evaluate has increased without a corresponding increase in the judgment's capacity to evaluate it.

Understanding remains bounded. Simon distinguished between information possession and information integration. Possessing information means having access to it. Integrating information means grasping how it fits together — seeing the architecture, feeling the structure, recognizing the patterns that connect disparate facts into a coherent picture. AI provides information possession at scale. Understanding — the felt sense of how a system works, the architectural intuition that recognizes when something is wrong before the wrongness can be articulated — is not a possession. It is a cognitive achievement, built through the kind of sustained, effortful engagement with a domain that deposits the pattern-recognition layers Simon and Chase documented in their chess research. Understanding accumulates slowly because the biological substrate on which it is built has temporal requirements that no tool can compress.

The capacity to define purpose remains bounded. This is the constraint that the preceding chapters have circled around from multiple angles — the problem-solving chapter's distinction between search heuristics and goal heuristics, the ant-on-the-beach chapter's distinction between the agent's rules and the environment's complexity, the design chapter's insistence that the builder must specify what success looks like before the AI can help achieve it. Purpose-setting is the most bounded of all cognitive operations, because it requires the integration of considerations that extend beyond any single domain: not just what is technically feasible, not just what the market demands, not just what the organization needs, but what is worth doing — what deserves to exist in the world, what serves the people it claims to serve, what justifies the resources it consumes.

AI cannot define purpose because purpose is not an information problem. It is not a computation problem. It is not a pattern-recognition problem that a sufficiently large training set can solve. Purpose emerges from the condition of being a bounded agent with stakes in the world — a creature that must choose how to allocate finite attention, finite time, finite life among the infinite possibilities that the world presents. A being with unbounded resources would not need to define purpose, because it could pursue everything simultaneously. Purpose is a consequence of boundedness — the recognition that you cannot do everything and therefore must decide what you will do, and the further recognition that this decision carries moral weight, because the thing you choose to build displaces the things you chose not to build.

Segal's question — "Are you worth amplifying?" — is, in Simon's vocabulary, a question about the quality of these bounded capacities. Not about the quality of the unbounded capacities that the AI provides. Those are given. They are the same for every builder who has access to the tool. The differentiator is the bounded residual: the attention that directs the tool, the judgment that evaluates its output, the understanding that recognizes when the output is genuinely adapted to the problem and when it merely resembles adaptation, the purpose that determines whether what gets built deserves to exist.

The bounded residual is where all the value is.

This claim may sound like consolation — a comforting story told to bounded minds about the irreplaceability of their limitations. Simon would have rejected the sentimental reading entirely. The claim is structural, not sentimental. It follows from the logic of the framework with the same necessity that satisficing follows from bounded rationality. When generation is unbounded, the only variable that differs between builders is evaluation. When implementation is free, the only variable that differs between products is the judgment that directed the implementation. When breadth is universal, the only variable that differs between practitioners is the depth that informs their choices.

The bounded residual is not valuable despite the unbounding of everything else. It is valuable because of it. The scarcity that determines value has shifted from the capabilities that AI provides to the capabilities it does not. And the capabilities it does not provide are the ones that have always been the most consequential determinants of whether what gets built serves people well or merely exists in the world consuming resources.

Simon would have recognized the current moment as a vindication of the insight he spent his career developing. Not a vindication of the specific predictions he made about AI's timeline — those were notoriously optimistic. Not a vindication of the physical symbol system hypothesis — the dominant AI paradigm has moved away from the symbolic processing he championed. But a vindication of the deeper insight, the one that earned the Nobel Prize: that the quality of what bounded minds produce depends not on the tools available to them but on the structures that channel their bounded capacities toward the decisions that matter.

The tools are now extraordinarily powerful. The structures are not yet adequate. The organizational attention architectures are being improvised rather than designed. The educational curricula are catching up rather than leading. The individual practices that distinguish builders who use AI wisely from builders who are used by it are being discovered through trial and error rather than through the rigorous study of design that Simon advocated.

The question is not whether AI will transform the relationship between human beings and their work. That transformation is already underway, observable in every metric and every testimony that You On AI documents. The question is whether the transformation will be designed or merely suffered — whether the architectures that manage the bounded residual will be built with the deliberateness that Simon's science of the artificial demands, or whether they will be left to emerge from the default dynamics of a system in which unbounded generation overwhelms bounded evaluation, and the appearance of capability substitutes for its substance.

Simon would have insisted — did insist, in every book and paper and lecture of his six-decade career — that design problems do not solve themselves. They require the intervention of bounded minds making choices about how things should be, and accepting the consequences. The AI age has produced the most consequential design problem in the history of human institutions: how to structure the interaction between bounded human minds and unbounded artificial capabilities so that the result serves human purposes rather than merely producing human output.

The bounds remain. They are not deficiencies to be mourned or limitations to be transcended. They are the architecture of human value — the constraints that make judgment necessary, attention precious, understanding hard-won, and purpose irreducibly personal. Everything that AI provides is now abundant and therefore common. Everything that remains bounded is now scarce and therefore decisive.

The science of the artificial has always been the science of design for bounded agents. Its moment has arrived.

Related You On AI Encyclopedia Topics for This Chapter

8 related entries — click to explore the full topic catalog

+

Every one of the 8 Orange Pill Wiki entries this chapter links to — the people, ideas, works, and events it uses as stepping stones. Click any card for the full entry.

Concept (8)

Concept

Attention as Binding Constraint (Simon)

Concept

Cognitive Surplus

Concept

Connoisseurship

Concept

Deliberation

Concept

Designing for the Bounded Builder

Concept

Organizational Inertia

Concept

Punctuated Equilibrium

Concept

The Judgment Economy

Epilogue

The satisficing threshold is the detail from Simon that I cannot put down. Not the attention scarcity, though that keeps me honest every time I sit down with Claude at two in the morning and notice I have been evaluating output on autopilot for the last forty minutes. Not the ant on the beach, though that metaphor has made me permanently suspicious of my own sense of competence when I am generating fast and evaluating slow. The satisficing threshold. The idea that "good enough" is not a fixed line but a floating one, and that the line moves every time the cost of trying again changes.

I felt it move in Trivandrum. I watched twenty engineers discover, inside a single week, that they could produce at a rate that would have been absurd the week before. The excitement in the room was genuine and earned. But what Simon's framework forced me to see is the thing I could not see while I was inside the excitement: the threshold was rising faster than the judgment that polices it. The engineers were generating more. Were they evaluating more wisely? I did not ask at the time. I was too busy being amazed. Simon would not have been too busy.

Here is why bounded rationality matters more now than when Simon formulated it in 1955: the bounds he identified have not changed, but the penalty for ignoring them has become catastrophic. When generation was expensive, bounded evaluation was tolerable. A builder who evaluated imperfectly still produced slowly enough that the imperfections had time to surface before the product shipped. The old friction, the implementation friction, was also an evaluation buffer. It gave you time to notice what was wrong while you were waiting for what was right.

That buffer is gone. The builder who evaluates imperfectly now produces at a rate where imperfections compound before anyone can catch them. The twenty-fold multiplier is real. It multiplies everything, including the consequences of every judgment call the builder makes. Good judgment, multiplied twenty times, produces extraordinary results. Mediocre judgment, multiplied twenty times, produces extraordinary volume of output that looked acceptable at each individual decision point and adds up to something nobody intended.

This is why I keep coming back to Simon's insistence that design is a form of knowledge. Not the design of the AI. The design of everything around the AI — the organizational structures, the evaluation rhythms, the educational investments, the personal disciplines that determine whether a bounded mind uses an unbounded tool wisely or is overwhelmed by it. Those designs are the dams in the river. Simon did not use that metaphor. He would have objected to its imprecision. But the function is the same: structures built by finite creatures to channel forces larger than themselves toward outcomes that serve life rather than merely generating activity.

My son's question at dinner — whether AI was going to take everyone's jobs — is a question about the left column. What AI unbinds. What it makes possible. It is the question everyone asks first, because the left column is visible, dramatic, and measurable. But Simon spent his career on the right column. What remains bounded. What no tool expands. What determines the quality of everything the left column produces.

The answer to my son, the honest answer, is that the jobs will change in ways I cannot fully predict. But the capacities that will matter most in whatever comes next are the ones that remain bounded: the attention to direct the tools, the judgment to evaluate their output, the understanding to recognize when something that works is not the same as something that is right, and the purpose to decide what deserves to be built at all.

Those capacities are not given. They are built. Slowly, through the specific, irreducible, often uncomfortable process of learning to think well in a world that does not make thinking easy. Simon knew this. He studied it for sixty years. The science of the artificial is, at bottom, a science of respect for finite minds and the structures they need to flourish.

We are finite. The tools are not. The design of the space between those two facts is the work of a generation.

-- Edo Segal

Related You On AI Encyclopedia Topics for This Chapter

4 related entries — click to explore the full topic catalog

+

Every one of the 4 Orange Pill Wiki entries this chapter links to — the people, ideas, works, and events it uses as stepping stones. Click any card for the full entry.

Concept (4)

Concept

Human Capital

Concept

Tacit Knowledge

Concept

The Fourth Bound

Concept

The Satisficing Threshold