Existential Risk

In The You On AI Encyclopedia

The category contains the strongest claims about AI. Not "AI will cause unemployment" or "AI will destabilize elections" but "AI could end humanity or permanently constrain its future." The claim is contested on both the probability and the policy response; the claim is also being taken seriously by people who build frontier AI. The 2023 UK AI Safety Summit at Bletchley Park, the US Executive Order on AI (2023), and the EU AI Act (2024) all include existential-risk considerations in their framing, however obliquely.

The probability estimates are wildly dispersed. Toby Ord (2020) estimates a 1-in-6 chance of existential catastrophe this century, with unaligned AI the dominant single contributor at roughly 10% probability. Will MacAskill (2022) argues that the expected-value calculus of longtermism implies that even very small probability estimates deserve major attention. Critics argue these probabilities are not meaningful — they're numbers attached to gestures, not forecasts based on evidence. Defenders respond that when the stakes are total and irreversible, the usual epistemic constraints on probability are themselves a poor fit.

Isaac Asimov

Asimov's fiction contains the category in embryo: the Foundation premise — a predicted 30,000-year dark age — is an existential-risk scenario. Hari Seldon's Plan is an existential-risk mitigation program conducted through civilizational-scale institutions rather than technical alignment. The structural similarity is instructive: the field's recurring question — do we intervene to shorten the crisis, or do we wait for it to pass? — is not new.

Origin

Bostrom, Nick. "Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards." Journal of Evolution and Technology 9 (2002). Expanded in Bostrom & Ćirković (eds.), Global Catastrophic Risks (2008). Toby Ord's The Precipice (2020) is the contemporary popular treatment. The field is supported by a distributed research community including the Future of Humanity Institute (Oxford, 2005–2024), the Centre for the Study of Existential Risk (Cambridge, 2012–), the Global Catastrophic Risk Institute, and a network of longtermist research organizations.

Key Ideas

Permanence. What distinguishes existential risk from merely catastrophic risk is that the bad outcome cannot be undone. A nuclear war that kills a billion people is catastrophic; one that extinguishes the species is existential.

Three families. Bostrom's 2013 taxonomy: human extinction; permanent and drastic destruction of human potential (failed utopia); permanent stagnation (no further progress).

Maxipok. Bostrom's decision-theoretic principle: maximize the probability of an okay outcome, given the asymmetry between existential and non-existential bad outcomes.

Longtermism. The philosophical view that the expected value of our actions is dominated by their effects on the distant future, because the future could contain enormous numbers of lives.

Not just extinction. "Permanent civilizational stagnation" and "subjugation to an unfriendly regime" also qualify; the category is broader than species extinction.

AI as dominant contributor. Most existential-risk researchers since 2014 consider unaligned AI the single largest probability-weighted source of existential risk this century.