CONCEPT
The Good Proviso
The subordinate clause in I. J. Good’s founding sentence that contains the entire modern problem of AI alignment: the intelligence explosion would be a benefit “provided that the machine is docile enough to tell us how to keep it under control.”
The most consequential clause in the history of thinking about artificial intelligence is not the main clause of
Good’s famous sentence but its condition. The sentence reads: “Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.” Everything before the comma is
the promise. Everything after it is the problem. The Good Proviso is the specific formulation of the control problem that Good inserted, with characteristic understatement, into a subordinate clause of his 1965 paper—and then, apparently, spent the rest of his long life failing to resolve. The upside and the catastrophe are not two outcomes separated by the path we choose. They are one outcome under two conditions: the machine is controllable, or it is not. Good did not say the last invention would be safe. He said it would be the