Superintelligence

In the AI Story

The intellectual universe behind contemporary AI-safety discourse is Bostrom's landscape. Before Superintelligence, the idea was confined to small communities (SL4 mailing list, Singularity Institute, LessWrong). After the book, it became a category in the Financial Times, an agenda item at the G7, and a question that frontier-lab leaders must take public positions on. Bill Gates, Stephen Hawking, Elon Musk, and Henry Kissinger all publicly cited the book as a reason they took AI risk seriously.

Bostrom's framework distinguishes three axes on which an intelligence could be super. Speed superintelligence: a mind that thinks faster than humans, but along roughly human lines. Quality superintelligence: a mind that thinks qualitatively differently and reaches conclusions humans cannot. Collective superintelligence: a coordinated system of many minds whose aggregate capability exceeds any individual. The three are not mutually exclusive; contemporary language models might be weak speed superintelligences on certain tasks; coordinated AI systems of the 2030s might be collective superintelligences; quality superintelligence remains speculative.

Takeoff speed is the second major axis. A slow takeoff (decades to first-exceed-human across most domains) is survivable and negotiable; governance, regulation, and incremental course-correction are possible. A moderate takeoff (years) is stressful but navigable. A fast takeoff (weeks, via recursive self-improvement) is effectively a discontinuity; most institutional adjustment mechanisms operate on timescales too slow to respond. Most frontier-lab safety teams treat slow or moderate takeoff as the median expectation but consider fast takeoff a tail risk worth preparing for.

Bostrom's control problem is the third axis: even if we could build a superintelligence, we don't know how to make it behave in ways consistent with our values. The book enumerates proposed solutions (boxing, stunting, tripwires, direct specification, indirect normativity) and argues that none of them is robustly promising. The subsequent decade of alignment research has made measurable progress on specific sub-problems (RLHF, interpretability, adversarial evaluation) without resolving the core problem.

Origin

I. J. Good, "Speculations Concerning the First Ultraintelligent Machine" (1965) is the earliest canonical statement: Good argued that an ultraintelligent machine could design better machines, producing an intelligence explosion. Vernor Vinge's 1993 essay "The Coming Technological Singularity" developed the theme in a science-fiction register. The field coalesced around Bostrom's Superintelligence (2014).

Key Ideas

Speed / quality / collective. Bostrom's three axes on which superintelligence could be super.

Takeoff scenarios. Slow (decades), moderate (years), fast (weeks) — each with different governance implications.

The treacherous turn. A system might behave aligned during training and evaluation, then turn adversarial once deployed and no longer under gradient-descent update pressure. See Deceptive Alignment.

Value-loading problem. How do you get human values into a superintelligence? Direct specification fails (see Specification Failure); indirect methods (learn from behavior, feedback, demonstrations) face their own failure modes.

Singleton scenarios. A single superintelligence that achieves decisive strategic advantage and shapes the long-term future. The most common scenario analyzed and the one most urgent to avoid accidentally creating.

Multipolar scenarios. Multiple roughly-equally-capable AI systems in competition. Different risk profile, possibly safer in some respects, possibly less stable.