PERSON
Stuart Russell
The man who wrote the textbook on AI and then told the field it had been building intelligence the wrong way from the start—architect of the control problem and of provably beneficial machines.
Stuart Russell is the rarest sort of critic: the one who built the thing he now warns against. With Peter Norvig he wrote
Artificial Intelligence: A Modern Approach, the textbook from which most working researchers first learned what the field even was. Then he stood up to say its foundation is cracked. The crack has a name—he calls it the standard model—in which we build machines as optimizers: we specify an objective, feed it into the machine, and unleash a capable optimizer upon it. The danger is not that machines will become evil. The danger is that they will become competent at pursuing objectives we did not specify carefully enough, and a sufficiently capable machine pursuing a fixed objective will pursue it past the point where we wanted it to stop. His remedy is to build machines that are, by construction, uncertain about what humans want—machines that defer, that ask, that welcome
the off switch as information rather than resisting it as defeat.