CONCEPT
Existential Risk
A category of risk whose realization would either annihilate humanity or permanently and drastically curtail its potential. AI joined this category in mainstream academic usage in 2014.
Existential risk is the academic category for risks whose realization is either terminal for humanity (extinction) or would lock in a permanently diminished future (lock-in of values, permanent stagnation, permanent subjugation). The category was formalized by Nick Bostrom in 2002 and elaborated by Toby Ord in The Precipice (2020). Canonical examples include asteroid impact, engineered pandemics, nuclear war, climate tipping points, and — since Bostrom's Superintelligence (2014) — unaligned artificial intelligence. The field is young, contested, and uncomfortable to study; it is also increasingly the frame within which AI policy is debated in Washington, Brussels, and London.
In The You On AI Field Guide
The category contains the strongest claims about AI. Not "AI will cause unemployment" or "AI will destabilize elections" but "AI could end humanity or permanently constrain its future." The claim is contested on both the probability and the policy response; the claim is also being taken seriously by people who build frontier AI. The 2023 UK AI Safety Summit at Bletchley Park, the