Second-Order Cybernetics

In the [YOU] on AI Field Guide

The cycle’s foundational argument—that you, the reader, are the missing variable in every account of the machine—is what second-order cybernetics makes rigorous. The first-order question—“how smart is this model?”—has no clean answer because the intelligence is a property of a human-machine interaction, not of the machine. This reframing is uncomfortable because it closes off two comforting positions at once: it denies the critic the satisfaction of dismissing AI as “mere prediction,” because the prediction is useful or not in proportion to the human who steers it; and it denies the enthusiast the convenience of attributing capability to the model alone, since a clumsy user running an impressive model produces clumsy outputs.

Second-order cybernetics also supplies the framework for the cycle’s account of human oversight. To be genuinely in the loop—in Pask’s sense—requires that the human can actually model the system: inspect its state, form hypotheses about its behavior, steer it in response to feedback, and be made more capable by the interaction over time. A person who nominally reviews AI outputs but cannot understand the system’s reasoning, cannot steer it, and is not improved by the oversight is not in the loop in any second-order sense. They are first-order theater, a spectator misidentified as a participant.

Origin

The doctrine emerged from the American cybernetics tradition founded by Norbert Wiener, which Pask encountered and extended through a series of cybernetics congresses beginning in the late 1950s. His meeting with Heinz von Foerster—director of the Biological Computer Laboratory at the University of Illinois—at a congress in Namur in 1958 was decisive. Von Foerster had been developing what he called the cybernetics of cybernetics: the application of the discipline’s concepts to the act of knowing itself. Pask brought to this the specific argument from conversation theory: that the scientist studying a learning system is not exempt from being a participant in it, because the observation alters the observed and the observer is altered in turn. Together they articulated the position that the boundary between observer and observed is not given by nature but drawn by the observer—a different observer might draw it differently, making observer-dependence not a limitation to overcome but a structural feature of every honest account.

Key Ideas

The observer is a term in the equation. Every description of a system embeds the describer’s choices—about what counts as the system’s boundary, what counts as its state, what counts as a meaningful measurement. These choices are not neutral; they shape what the system is, as described. For AI, this means every performance claim is a claim about a context of use, not about a Platonic model-in-itself.

Observer-dependence is not relativism. Second-order cybernetics does not say anything goes. There is still a difference between a skilled and an unskilled use of an AI system, still a difference between a true and a false claim about what an interaction produced. What changes is that these differences cannot be cashed out as properties of the model alone. The observer is not free to see anything; the observer is, however, ineliminably part of what there is to see.

Benchmarks as interactions. Scaling law benchmarks and capability evaluations measure the performance of a model on a particular evaluation procedure designed by particular humans. Second-order cybernetics treats the celebration of benchmark results as a first-order error: the score is a property of the system that includes the benchmark, not of the model alone. This is not a call to abandon evaluation but a call to design evaluations that honestly include the human context of use.

System closure and the design of good loops. A second-order perspective reframes AI design from “make the model better” to “make the loop better.” Retrieval, tool use, agentic feedback, and the design of interfaces that expose rather than conceal the model’s reasoning are all moves toward a tighter, more honest loop—one in which the human is a genuine participant rather than a receiver. The emphasis on interpretability in AI safety research is, from this angle, an attempt to give the human observer a real position in the system rather than a nominal one.