The Satisficing Trap

In the AI Story

The satisficing trap operates through a precise mechanism. The builder has a criterion — the code must work, the design must be functional, the analysis must be coherent. The AI generates an output that meets the criterion. The builder, rationally operating under time pressure and cognitive constraints, accepts the output and moves forward. The acceptance is defensible at every step, yet its cumulative effect is that the builder never encounters the alternatives that would have required further search — the unconventional approaches, the experimental techniques, the solutions that lie beyond the AI's statistical comfort zone.

What makes the trap particularly resistant to intervention is that the AI's outputs are not merely adequate but often genuinely good. They represent the distilled conventional wisdom of the training corpus. A builder who holds out against satisficing — who refuses the AI's competent output and continues searching — must be able to articulate a specific reason for the refusal, and most of the time she cannot, because the output does meet her stated criteria. The refusal would be an act of principle rather than a response to specific inadequacy.

The trap compounds across time. Each instance of satisficing trains the builder's expectation of what "satisfactory" looks like, and the expectation shifts toward the AI's center of gravity. Over time, the builder's criteria themselves become calibrated to what the AI produces, making the gap between "satisfactory" and "optimal" invisible. The trap is not only that she stops searching; it is that she stops knowing what she would be searching for.

Pariser's prescription is not to eliminate satisficing — which would be cognitively impossible — but to introduce structural features that interrupt the trap at specific points. A divergence prompt that deliberately produces an output far from the builder's request is an interruption: it reminds the builder that the convergent output was a selection, not the totality. An assumption surface that articulates the implicit criteria in the builder's prompt is another: it makes visible what the satisficing is satisficing against.

Origin

Simon's original formulation appeared in "Rational Choice and the Structure of the Environment" (1956) and was developed across his career in work on bounded rationality. Its application to AI-augmented production emerges from the recognition that generative systems, by design, optimize for adequacy relative to user intent — which is precisely the condition that makes satisficing operationally automatic.

Key Ideas

Satisficing is rational and structurally exploitable. The decision strategy that makes humans functional under constraints is the same strategy that AI systems can exploit to foreclose the search for alternatives.

Good outputs are more dangerous than mediocre ones. Mediocre outputs trigger continued search; genuinely good outputs that meet criteria stop the search.

Criteria shift toward the center. Over time, the builder's expectation of "satisfactory" calibrates to what the AI produces, making the gap between satisfactory and optimal invisible.

Interruption must be structural, not motivational. Willpower alone cannot sustain refusal of competent outputs; the workflow architecture itself must introduce variance.