Runaway Coevolution

In the AI Story

Runaway coevolution is distinguished from healthy coevolutionary relationships by a specific feature: the feedback lacks corrective mechanisms. In normal coevolution, changes that take the system too far in one direction produce consequences that push back — the male with the tail so elaborate he cannot fly is eaten, the preference for such tails declines, equilibrium returns. In runaway coevolution, the corrective mechanisms are absent or are themselves pulled into the positive feedback loop, so the system cannot self-correct.

The human-AI relationship exhibits runaway features. Engineers who work with AI become more skilled at AI-mediated work and less practiced at unmediated work. Their declining capacity for unmediated work makes them more dependent on AI. Their dependence produces market pressure for better AI, which makes them still more effective at AI-mediated work and still less practiced at the alternative. The trajectory does not require anyone to intend it. It emerges from the feedback structure itself.

The institutional dimension is similar. Companies that adopt AI faster than competitors gain short-term advantage. Competitors must respond or fall behind. The arms race pulls entire industries toward faster adoption than any firm would choose individually if it could choose without reference to competitors. The runaway operates at the industry level, not just the individual level.

Arresting runaway coevolution requires introducing negative feedback at a level that can modulate the positive feedback loop. At the individual level, this might mean deliberate practice in unmediated work — monk mode, deliberate non-device time. At the institutional level, it might mean norms and structures that reward quality over speed, depth over throughput. At the systemic level, it requires institutions that operate outside the competitive pressure of the arms race — the dams that redirect flow rather than trying to stop it.

Origin

Bateson developed the concept of runaway in the context of his broader cybernetic framework. The peacock's tail example — drawn from R.A. Fisher's 1930 analysis of sexual selection — was his favored illustration of positive-feedback coevolution. The framework extended to cultural evolution, where he argued that certain cultural practices (arms races, status competitions) exhibit the same runaway dynamics.

Contemporary extensions appear in work on task seepage, burnout society, and the dynamics of attention capture. The common thread is that positive-feedback dynamics produce trajectories no individual chose and none can exit without coordinated intervention.

Key Ideas

Positive feedback produces runaway. When each party's changes amplify rather than correct the other's, the system spirals toward extremes neither chose.

Corrective mechanisms must be preserved. Healthy coevolution has negative-feedback mechanisms that limit excursions from equilibrium; runaway occurs when these are absent or captured.

Human-AI coevolution exhibits runaway features. Increasing dependence and increasing capability feed each other in ways that trend toward tighter coupling.

The institutional arms race is the same pattern at scale. Competitive pressure pulls industries toward adoption rates no firm would choose individually.

Arrest requires new negative feedback. Individual willpower is insufficient; structural interventions that operate outside the positive-feedback loop are required.