CONCEPT
Structure Over Intention
Oppenheimer’s hard-won insight, confirmed by every episode in the atomic age, that the
trajectory of a powerful technology is determined by the configuration of incentives and competitive pressures the technology creates—not by the intentions of the people who build it, however good those intentions are.
Oppenheimer wanted international control and got an arms race. He argued for candour and was destroyed for it. The General Advisory Committee, the most distinguished group of physicists in America, recommended against a crash hydrogen bomb program and was overridden in weeks. Every time his intentions and the intentions of people like him collided with the structure of the competitive situation the bomb had created, the structure won. His central, uncomfortable lesson is not that intentions are worthless but that they are radically insufficient: that even with conscientious builders, careful institutions, and well-meaning officials, the structure of competition can drive a technology toward dangerous outcomes that no one intended and everyone, individually, might have preferred to avoid. The
AI governance conversation is overwhelmingly a conversation about intentions—getting the right values into the systems, cultivating responsible cultures, keeping the technology out of bad hands—and Oppenheimer’s life insists that this conversation is