Herbert Simon

In the [YOU] on AI Field Guide

Simon enters the cycle at the point where the triumphalist account of AI as an unbinding of human cognitive constraints requires its most careful examination. The cycle documents a genuine unbinding: builders who can now produce at speeds and scales that would have been unimaginable five years earlier, crossing disciplinary boundaries that previously required years of specialized training. Simon's framework accepts this unbinding and then asks the question the triumphalist account forgets to ask: which bounds have been relaxed, and which remain? His answer, written in 1971 before any AI tool existed, is devastating in its precision. Information, computation, and time—the three constraints that produced bounded rationality—have been radically relaxed by AI. The fourth constraint, which Simon identified as the binding one in any information-rich environment, has not been relaxed at all. Attention—the cognitive resource required to evaluate what the computation has produced, to judge whether the generated alternative is actually good enough—remains as bounded as it was before the machine arrived.

The evaluation bottleneck is Simon's contribution to the cycle's central argument. When generation was expensive—when building a software prototype took months, not minutes—the binding constraint was what could be created. The satisficing threshold was calibrated to a world where the next alternative cost real time and effort to produce, and the threshold was accordingly modest. When generation becomes nearly free, the threshold rises: the builder can afford to demand more, to evaluate more, to reject more. But evaluation costs remain constant, and the threshold keeps rising while the evaluative capacity that must meet it does not. The result, documented by Berkeley researchers and named by the cycle, is a specific exhaustion that is not overwork in the traditional sense but unbounded satisficing—a search that no longer terminates because the cost of the next iteration approaches zero and the threshold recedes like a horizon.

Simon's concept of choice architecture—his demonstration, in decades of organizational research, that the structure of a decision environment shapes decisions more reliably than the preferences of the decision-makers who operate within it—gives the cycle its sharpest tool for understanding what AI tools do that their users cannot easily see. Every AI response is a choice architecture. It presents certain alternatives and filters others, structures the problem in a particular way, signals confidence in a format that bounded minds interpret as authority. The builder evaluates the output within the frame the AI has constructed, and the frame, by its nature, cannot be fully seen from inside it. The most experienced builders—those with deep enough pattern libraries to recognize when the AI has filtered out something important—are the least vulnerable to this effect. The least experienced builders, who most need exposure to alternatives they have not previously considered, are the most thoroughly shaped by the architecture they cannot see.

Origin

Herbert Simon was born in Milwaukee in 1916 and trained in political science at the University of Chicago, where his doctoral dissertation on administrative decision-making contained, in embryonic form, the argument that would eventually earn him the Nobel Prize. He was not a psychologist, not an economist, not a computer scientist—he was a political scientist who had noticed that every formal model of decision-making assumed a decision-maker who did not exist. The perfectly rational agent of economic theory, who gathered all available information, evaluated every possible alternative, and selected the utility-maximizing option, was not a simplification of the real decision-maker. It was a fiction so removed from the real decision-maker that building institutions on its assumptions produced institutions that consistently failed.

His intervention, in the 1955 paper 'A Behavioral Model of Rational Choice,' was to describe the species that actually makes decisions: a bounded agent who evaluates alternatives sequentially, applies a threshold of acceptability, and stops searching when the threshold is met. The word he coined for this process—satisfice, a portmanteau of satisfy and suffice—entered the social sciences as a technical term and remains there, one of the few concepts from that era that has survived both empirical scrutiny and theoretical challenge. He spent the following decades tracing the architectural implications: if all human decision-making is bounded in this way, then every institution, organization, and tool ever built by human beings is, at bottom, a device for managing those bounds. The design challenge is not to eliminate bounded rationality but to build structures that channel it toward wisdom rather than waste.

Key Ideas

Bounded rationality. Simon's foundational thesis: human decision-makers operate under binding constraints of information, computation, and time, and the rational response to these constraints is not optimization but satisficing. Bounded rationality is not a defect to be corrected; it is the condition of any finite mind in a complex environment, and every institution ever built is a response to it. AI relaxes three of the four constraints that produce it while leaving the fourth—attention—untouched.

Attention as the scarce resource. 'A wealth of information creates a poverty of attention.' Simon stated this in 1971, two decades before the world wide web and five decades before generative AI. The principle is not about distraction in the colloquial sense; it is about the metabolic relationship between information and cognition. Information consumes attention; more information does not help a bounded mind if the bound is attention rather than information. Attention as the binding constraint is the principle that the current discourse about AI productivity consistently fails to apply.

Satisficing and the threshold. The satisficing threshold—the criterion of 'good enough' that bounded agents carry into every decision—adjusts to the cost of search. When search is cheap, the threshold rises; when search is expensive, the threshold falls. AI makes generation nearly free, which drives the threshold upward without expanding the evaluative capacity needed to reach it. The result is the evaluation bottleneck: bounded evaluation meeting unbounded generation.

Choice architecture. The structure of a decision environment shapes decisions more reliably than the preferences of the decision-makers. Simon documented this in organizational hierarchies; the principle applies directly to AI tools, which present certain alternatives, filter others, and frame problems in ways the builder cannot fully see from inside the interaction. Choice architecture—the design of decision environments to channel bounded attention toward better outcomes—is Simon's gift to AI ethics: not a question about values but a question about structure.

The science of the artificial. In The Sciences of the Artificial (1969), Simon proposed that designed things deserve their own science—the study of how artificial systems interface between their inner logic and the outer environments they must serve. The core concept, the inner/outer environment distinction, applies directly to AI: the system's inner environment (its training, its architecture, its weights) must interface with the outer environment (the world it must serve), and the interface must be managed by a bounded builder whose attention is the scarce resource. The quality of everything built in the AI age depends on the quality of that interface management.