Feedback, Not Control

In the AI Story

The distinction matters for how we conceive the human role. If AI is a control system, the human's job is to specify goals and let the system execute. If AI is a feedback system, the human is a participant in a circuit whose continuous evaluation is constitutive of the output's quality. Bateson insists on the second model because it is a more accurate description of how the system actually works. 'Feed it genuine care and it carries that further than any tool in human history' is a statement about feedback, not control.

In healthy cybernetic systems, feedback is negative — it corrects deviations and returns the system to equilibrium. The thermostat exemplifies this. In pathological systems, feedback is positive — it amplifies deviations, producing runaway. The microphone screech is the canonical example. The Berkeley study of AI-augmented workers documents exactly this pathology: task completion produces satisfaction which motivates the next task, which the AI makes immediately available, which reduces friction between impulse and action, which accelerates the cycle. Task seepage is a classic positive-feedback signature.

Individual metacommunicative interventions — asking oneself 'am I here because I choose to be or because I cannot leave?' — are negative-feedback mechanisms, but they depend on the individual's willingness to interrupt a process that by its nature resists interruption. Bateson would insist that structural negative feedback is required: mechanisms built into the circuit rather than dependent on individual willpower. Structured pauses, sequenced workflows, protected human-only time — these are circuit-level governors.

The beaver metaphor captures the structural point precisely. The beaver does not appeal to the river to slow down; it builds a dam, a physical structure that modifies watershed feedback dynamics. The dam converts destructive positive feedback into regulated negative feedback. Governance is cybernetic construction, not moral exhortation.

Origin

Bateson's reframing of cybernetics emerged from his disappointment with how the field's popular reception had distorted its core insight. In papers collected in Steps to an Ecology of Mind and in his dialogues with Norbert Wiener, Claude Shannon, and Warren McCulloch at the Macy Conferences, he insisted that the field's contribution was to replace linear thinking with circular thinking — not to provide new tools for old forms of control.

Contemporary cybernetic thinkers including Francisco Varela and Heinz von Foerster extended this insight into 'second-order cybernetics' — the cybernetics of systems that include the observer. Second-order cybernetics makes explicit what Bateson had always insisted: the designer of a feedback system is part of the system, not outside it.

Key Ideas

Feedback is circular; control is linear. The cybernetic insight is that causation in living systems flows through loops, not chains.

Quality is an emergent property of the full loop. No single component — not the AI, not the human — determines output quality alone.

Negative feedback corrects; positive feedback amplifies. The pathology of AI-augmented work is positive-feedback runaway masquerading as productivity.

Structural governors beat willpower governors. Individual metacommunicative practice is necessary but insufficient; the circuit needs governors built into its architecture.

Second-order feedback matters. The tools evolve through how they are used; cultural patterns of use shape tool development, which shapes culture.