DARPA Explainable AI Program

The DARPA Explainable Artificial Intelligence program, launched in 2016, addressed what DARPA leadership identified as one of the most significant barriers to effective deployment of AI systems: users did not understand how the systems worked, did not know when to trust them, and did not know how to detect failures. DARPA assembled eleven teams of AI researchers to build more explainable systems, and — in a decision that reveals something important about the program's philosophical sophistication — established a separate team of cognitive psychologists led by Klein, tasked with understanding what explanation actually means from the perspective of the humans who need it. The distinction between technical explainability and effective human oversight became the program's defining intellectual contribution, producing both the AIQ toolkit for user-centered assessment and a research literature on the cognitive requirements for appropriate AI trust.