Designing for the Gap

In the AI Story

The oracle model is the path of least resistance for AI product design. It is what users, in the short term, prefer. It is what organizations measure. It is what investors reward. It has a coherent metric — output quality — and a clear deliverable: the right answer, efficiently produced. By these measures it works, and working by these measures is what makes the path of least resistance feel like the right path.

The alternative philosophy starts differently. It asks what the user actually needs — not just an output but the capacity to act competently with the output, which requires situated knowledge the output does not contain. This reframing produces different design choices. An AI coding assistant designed for the gap might generate a scaffold rather than a complete implementation, asking the user to write the code that connects the components so she encounters the specific resistances that produce understanding. An AI writing assistant might produce questions rather than drafts, preserving the cognitive friction through which writers discover what they think. An AI tutor might withhold answers and support the student's own engagement with the problem, rather than delivering the solution.

The pedagogical application is most immediately actionable. The Orange Pill describes a teacher who stopped grading essays and started grading questions — asking students to produce the five questions they would need to ask before they could write an essay worth reading. The assignment preserves the cognitive friction of inquiry while allowing AI to assist with the mechanical production of answers. Suchman's framework names why the intervention works: it keeps the student in the gap, confronting what she does not understand, while treating AI as a resource within a practice that preserves the developmental work of inquiry.

The market will not demand these designs. Users prefer oracles in the short term. Organizations prefer efficiency. Investors prefer scale. The oracle model is easier to build, easier to sell, and easier to measure. Designing for the gap requires an institutional commitment to user development that the market's incentive structures do not naturally reward — a commitment to measuring not just what was produced but what the producer learned, not just whether the output is correct but whether the user is equipped to recognize when the next output is not. This is why the design question is ultimately an institutional question, not a technical one.

Origin

The philosophy has antecedents in Douglas Engelbart's augmentation vs. automation distinction, in Alan Kay's design for the Dynabook, and in the broader tradition of educational and participatory design that Suchman's work belongs to. It emerges most explicitly in contemporary debates about AI in education, pair programming, and clinical decision support, where the question of whether tools should replace or scaffold human judgment is increasingly pressing.

Suchman's framework provides the theoretical backing for the philosophy by making explicit what it preserves: the situated knowledge that develops only through practitioner engagement with the resistance of specific material reality.

Key Ideas

Oracle vs. scaffold. The oracle model optimizes for output; the scaffold model optimizes for user development.

Productive friction. Good scaffolding preserves the developmental friction through which situated knowledge accumulates.

Pedagogy as case. Teaching offers the clearest application — AI that supports inquiry rather than delivering answers preserves the learning the assignment is supposed to produce.

Visibility of reasoning. Designing for the gap means making AI's reasoning more visible, not less — so users can evaluate the process, not just accept the output.

An institutional commitment. The market will not produce these designs without institutional pressure. The question is whether institutions will invest in user development when efficiency is easier to measure.