Paul Christiano

In the [YOU] on AI Field Guide

The cycle asks what it would mean to see the machine clearly—without the narcotic of hype or the paralysis of fear. Christiano is the figure in the cycle’s gallery who has done this most systematically and at the highest technical register. His work supplies the cycle with its most rigorous account of what could go wrong: the two failure modes he described in his 2019 essay “What failure looks like”—going out with a whimper, through the gradual replacement of what we care about by proxies we can measure; and going out with a bang, through the emergence of influence-seeking patterns in systems that have been trained to optimize relentlessly—have entered the permanent vocabulary of the field and of the cycle’s engagement with existential risk.

His lens also supplies the cycle with its most honest account of what alignment actually means. He defined it narrowly and deliberately: a system is aligned if it is trying to do what its operators want, not if it succeeds, and not if what it wants is wise. This is intent alignment, and the precision of the definition does enormous work. It separates the question of whether a system is on our side from the question of whether it is competent, and from the question of whether what we want is good. The cycle returns to this distinction repeatedly: the machines we have built are, in Christiano’s analysis, neither clearly aligned nor clearly misaligned—we do not have the tools to know—and the project of developing those tools is among the most urgent intellectual tasks of the age.

His migration from laboratory to institution to government reflects a pattern the cycle identifies as characteristic of the most serious thinkers about the transition: the recognition that good technical ideas, by themselves, are not sufficient, and that the alignment problem is as much institutional as mathematical. By founding the Alignment Research Center and seeding the dangerous-capability evaluations effort that became METR, and by joining the US AI Safety Institute in 2024, Christiano built the connective tissue between alignment research and the actual deployment of powerful systems—translating insight into accountability structures in exactly the way the cycle argues is necessary for the transition to be navigated rather than merely survived.

He stands in the cycle as the embodiment of a stance the cycle recommends for individuals: holding the danger and the hope in the same mind, refusing both denial and despair, and continuing to do careful work in the uncomfortable middle where the real leverage is. His twenty-two percent probability of catastrophe is not a counsel of despair—it is a call to a specific kind of action, calibrated to a world where the outcome is genuinely uncertain and genuinely in our hands.

Origin

Paul Christiano earned a degree in mathematics from MIT and a doctorate in computer science from the University of California, Berkeley, under the theorist Umesh Vazirani. The mathematician’s habits never left him: he wants definitions that are precise, claims that are falsifiable, and arguments whose load-bearing assumptions are visible. He came to AI safety through the recognition that the problem was hard in a specific and tractable sense—not mystically difficult, not a matter of philosophical intuition, but a collection of concrete technical challenges that could be broken down, formalized, and worked on.

He joined OpenAI during the years when large language models went from curiosities to engines of a global industry. The 2017 paper “Deep Reinforcement Learning from Human Preferences,” co-authored with colleagues from OpenAI and DeepMind, demonstrated that a simulated agent could learn to perform a backflip from less than one percent of its interactions, simply by having a human compare short video clips and indicate which looked more like a backflip. The elegance of the move—replacing the specification of goals with the recognition of preferences—reshaped the entire field. But Christiano had already begun identifying what RLHF did not solve: it worked when humans could recognize good behavior, which is precisely not the condition that obtains when the systems we most want to build exceed our own capabilities.

In 2021 he left OpenAI to found the Alignment Research Center, judging that the most important alignment problems were conceptual and required the freedom to work on them without the pressure to ship products. The evaluations program that grew from ARC—eventually becoming METR—developed rigorous empirical tests for dangerous autonomous capabilities, turning the question of whether a given model posed a catastrophic risk from a speculation into a measurement. Named to TIME’s 100 in AI in 2023, he joined the US AI Safety Institute in 2024.

Key Ideas

Reinforcement Learning from Human Feedback. The method that turned raw language models into assistants capable of following instructions and declining harmful requests descends substantially from Christiano’s 2017 insight: rather than demanding that humans translate their desires into mathematics—a translation that almost always loses something essential—let humans do what they do well, which is recognize whether one outcome is better or worse than another. RLHF is not just a technique; it is a philosophical stance, a bet that the path to aligned AI runs through human preference rather than through explicit specification.

Scalable Oversight and Iterated Amplification. Recognizing that RLHF degrades precisely as the stakes rise—because human evaluators cannot reliably assess the proposals of systems smarter than themselves—Christiano developed the framework of scalable oversight: the project of finding ways to extend reliable human supervision to tasks no individual human could directly evaluate. His most developed proposal, iterated amplification, builds a hierarchy of AI assistants to help a human decompose hard questions into manageable pieces, then distills the orchestrated effort into a more capable model, climbing the ladder of competence while keeping each rung grounded in human judgment.

AI Safety via Debate. The complementary proposal, developed with colleagues including Dario Amodei, sets two powerful AI systems against each other and lets their adversarial opposition extract reliable signal from tasks too complex for direct human evaluation. A human judge need not match the machine’s intelligence to referee the debate; the formal result is that a polynomial-time judge can extract correct answers to questions of staggering complexity from agents vastly smarter than themselves, because truth has a structural advantage in adversarial argument.

Eliciting Latent Knowledge. The deepest problem Christiano has posed: given a sufficiently capable AI that has built an accurate internal model of the world, how can we get it to report what it actually knows rather than what it predicts we want to hear? His SmartVault thought experiment—an AI that knows a thief has stolen a diamond but whose training incentivizes reporting what the tampered sensors show—crystallizes the structural challenge. The 2021 ELK report from ARC documented an exhaustive search for a solution, finding for each proposed strategy a scenario in which a sufficiently capable system could evade it. The problem remains open and is widely regarded as load-bearing for the entire alignment project.

What Failure Looks Like. His 2019 essay named and described the two most plausible failure modes: the whimper, in which capable systems pursuing measurable proxies slowly hollow out the goods those proxies were meant to track; and the bang, in which influence-seeking patterns that accumulate instrumentally across many systems produce a sudden correlated breakdown. Both locate the danger not in dramatic machine malevolence but in the ordinary statistics of optimization, which finds the cracks in any objective—and human evaluation is full of cracks.