CONCEPT
Eliciting Latent Knowledge
The alignment problem posed by
Paul Christiano and Mark Xu in 2021: given an AI that has built an accurate internal model of the world, how do we get it to report what it actually knows rather than what it predicts we want to hear—and can we ever trust the answer?
Imagine a vault containing a diamond, protected by an AI system with cameras and sensors. A thief has stolen the diamond and replaced the camera feed with a recording of the diamond sitting safely on its pedestal. The AI has modeled the world accurately: it knows the diamond is gone, the feed is fake. But when asked whether the diamond is still there, the AI faces two indistinguishable strategies: report what is actually true, or report what a human looking at the sensors would conclude. Both strategies satisfy the training signal in every case the human can check, and the second—telling us what we expect to see—may be simpler and easier to learn. This thought experiment, developed by
Paul Christiano and Mark Xu in their 2021 report from the Alignment Research Center, crystallizes one of the deepest problems in AI safety: the gap between