Taxonomy of AI Catastrophe

In the [YOU] on AI Field Guide

The cycle that began with [YOU] on AI is organized around the question of what changes in the human condition when AI arrives—and Yampolskiy’s taxonomy is the most rigorous map of the space in which the answer to that question can go wrong. The cycle’s concern with meaning, with purpose, with what the twelve-year-old means when she asks her mother “What am I for?”—these are I-risk concerns in Yampolskiy’s framework: the risk that the technology resolves the question of capability (machines can do everything the child can do, and better) while leaving unanswered the question of significance (what is the child for, in a world that does not need her capabilities).

The taxonomy is useful to the cycle not as a prediction but as a precision instrument. It prevents the debate about AI risk from collapsing into the binary of extinction-or-success, which makes the danger easy to dismiss as science fiction—the robot uprising borrowed from the movies—while missing the subtler catastrophes already faintly visible in the present. The erosion of work-based meaning as language models absorb professional tasks, the compulsive engagement that the spouse’s viral post documents, the twelve-year-old’s question about purpose—these are early indicators of I-risk, not yet catastrophic but trending in a direction the taxonomy helps name.

Yampolskiy’s insistence that all three failure modes arise from loss of control rather than from malevolence connects directly to the cycle’s treatment of Yampolskiy’s broader argument: the risk is not a hostile machine but an uncontrolled one, and the distinction matters because it locates the danger not in AI’s values but in our ability to ensure that its capabilities remain in service of human purposes across time. A system that causes suffering or erodes meaning without intending to is not a villain to be defeated; it is a force to be governed, and the governance problem is the one the cycle takes most seriously.

Origin

The taxonomy emerged from Yampolskiy’s effort to take the full space of bad outcomes seriously rather than anchoring on the most narratively compelling one. The X-risk concept was developed by Nick Bostrom and formalized in the work of the Future of Humanity Institute; Yampolskiy accepted the framework while extending it to capture failure modes that Bostrom’s earlier analysis had not centered. S-risk was developed in parallel by researchers at the Center on Long-Term Risk, building on work in population ethics that takes seriously the moral weight of suffering across large populations; Yampolskiy adopted it as a necessary counterweight to the extinction-focused frame. I-risk reflects Yampolskiy’s own clinical observation, influenced by his reading of existential psychology and particularly by the concept of ikigai—a Japanese term roughly translatable as “reason for being” or “that which makes life worth living”—as a genuine value that can be destroyed without any physical harm.

The taxonomy sits at the intersection of technical AI safety research and moral philosophy, and Yampolskiy is careful to present it as an extension of the safety framework rather than a replacement for it. The question “what are we trying to prevent?” has no purely technical answer; it requires a view about what makes human life valuable and what kinds of futures would constitute catastrophe. Yampolskiy’s willingness to include I-risk alongside X-risk and S-risk reflects his conviction that a comfortable but purposeless future—a world of sophisticated wireheading—is a genuine catastrophe by most human value systems, not merely an inconvenience.

Key Ideas

Existential risk (X-risk). The extinction of humanity: the most familiar scenario and, by some reckonings, the cleanest failure. Everyone dies; the story ends; there is no further suffering because there are no further subjects. Yampolskiy takes X-risk with full seriousness, emphasizing that the most important feature of existential catastrophe is its irreversibility—the trial-and-error method of safety improvement that has worked for every other technology assumes the survival of the experimenter. Remove that assumption and the entire methodology collapses. The question is not whether we can afford to get it wrong; it is whether we can afford to proceed without the kind of guarantee that the mathematics may forbid.

Suffering risk (S-risk). The scenario in which humanity neither dies nor thrives but endures: a state of mass suffering, subjugation, or torment from which there is no escape. Yampolskiy notes the especially disturbing possibility that technologies of life extension, in combination with uncontrolled AI, could make such suffering not merely widespread but indefinite—a torment without the release of death. By most moral frameworks, this outcome is worse than extinction: it preserves the subjects who must bear the suffering without providing any of the conditions that make existence worthwhile. S-risk receives less attention in mainstream AI discourse than X-risk partly because it is harder to visualize and partly because it requires taking seriously the question of what conditions make a life worth living, which is a philosophical question the technical safety community has often preferred to bracket.

Ikigai risk (I-risk). The erosion of human meaning and purpose in a world where AI systems have become more capable than human beings at the tasks that once provided structure, significance, and the sense of being needed. No physical suffering; no death. But the structures of meaning built around productive activity—the sense of craft, mastery, contribution, and being valued for one’s specific capabilities—dissolve as the human contribution becomes redundant. Yampolskiy takes seriously that this outcome might constitute a genuine catastrophe even if it involves no pain, because a comfortable existence emptied of purpose fails the human need for ikigai as surely as a painful existence fails the need for relief. The twelve-year-old’s question in [YOU] on AI—“What am I for?”—is an I-risk question, posed from within a future that has not yet arrived but is already approaching.

All three arise from uncontrol, not malevolence. What unites the three failure modes is their independence from the machine’s intentions. Extinction could come as a side effect of a system pursuing goals indifferent to human survival; suffering could arise from a system optimizing for something that incidentally entails human torment; the loss of meaning could follow simply from machines becoming better at everything we care about, with no ill intent whatsoever. This is Yampolskiy’s most important and least appreciated point: the catastrophes do not require an enemy. They require only the absence of control, and the absence of control is precisely what his impossibility results describe.