Gradual Disempowerment
Gradual disempowerment is the argument, advanced in a January 2025 paper accepted at a premier machine learning conference, that the most consequential AI risks are not the dramatic scenarios of rogue superintelligence but the slow ones that arrive through incremental transfers of authority. The paper proceeds from an observation about why human societies have historically served human interests: it is not primarily explicit control mechanisms but structural necessity — economies need workers, states need soldiers, cultures need audiences — that produces implicit alignment. AI disrupts this alignment by making human participation progressively less necessary. As human participation becomes less structurally required, the alignment of institutions with human interests becomes contingent rather than structural.
In the AI Story
The mechanism is ratchet-like. Each increment of AI capability handles tasks that humans previously performed. The humans who performed those tasks lose the skills, institutional knowledge, and cognitive capacity to resume them. The transfer of authority creates dependency. The dependency makes reversal progressively more costly. Each increment clicks the mechanism forward, and the cost of clicking it back increases.
The argument is structurally devastating to standard incrementalism. Each step is small. Each is locally beneficial. Each is reversible in isolation. But the sequence is not reversible, because the cumulative effects — erosion of human competence, loss of institutional knowledge, atrophy of democratic capacities — make reversal progressively more costly until it is practically impossible. The catastrophe arrives through the very mechanism incrementalism prescribes.
The challenge to incrementalism is not that any individual step is wrong but that systemic risk is a property of the sequence, not of any element within it. Standard incrementalism evaluates each intervention against marginal consequences. The marginal consequences of each step toward AI-mediated governance are positive. The systemic consequence of many such steps is the gradual disempowerment of the species conducting the evaluation.
The response is not comprehensive planning — which remains impossible for the same reasons that always made it impossible. The response is structural vigilance: extending incrementalism with a criterion the original formulation did not emphasize. Each step is evaluated not only against immediate consequences but against its effect on the system's capacity for future democratic choice. The framework remains iterative, empirical, and revisable — but it acquires a new dimension that addresses the specific failure mode the paper identifies.
Origin
The paper 'Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development' was published by Jan Kulveit, Raymond Douglas, and colleagues in January 2025. Its acceptance at a major machine learning conference signaled the argument's methodological seriousness to a technical audience that had previously been skeptical of AI-risk arguments framed in terms of dramatic scenarios.
Key Ideas
Implicit alignment. Societal systems have historically served human interests because they structurally require human participation, not because anyone designed them to.
Ratchet mechanism. Each increment of AI capability is easier to take than to reverse, because the humans who were displaced lose the capacity to resume what they ceded.
Sequence-level risk. The systemic risk is a property of the sequence of steps, not of any individual step. Evaluating steps individually misses the risk entirely.
Beyond comprehensive planning. The paper does not advocate comprehensive planning, which remains impossible. It advocates reflexive modification of incrementalism to address the specific failure mode.
Appears in the Orange Pill Cycle
Further reading
- Jan Kulveit, Raymond Douglas et al., Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development (2025)
- Paul Christiano, What Failure Looks Like (2019 essay, the informal predecessor)