AI Uncontrollability

To control a system, argues Roman Yampolskiy, you must be able to do three things: understand what it is doing, predict what it will do, and influence what it will do next. Strip away any one of these and the word “control” becomes a courtesy rather than a fact. For a sufficiently advanced artificial intelligence, his unsettling thesis holds, all three may fail simultaneously and necessarily. The system’s reasoning may be unexplainable—any faithful account of it too complex for a human mind to receive, any simplification a distortion that the system’s actual reasoning did not match. Its behavior may be unpredictable—precisely because predicting what a smarter agent will do requires being as smart as that agent, so that prediction collapses into simulation and simulation of a superior mind is unavailable to an inferior one. And with understanding and prediction both failing, influence becomes a word without a referent: you cannot reliably steer what you cannot understand or anticipate. What