CONCEPT
Reflexive Prediction
Oskar Morgenstern's foundational insight that predictions about intelligent agents can destroy themselves by being made—that the predicted party, once it learns the forecast, can act to falsify it—a structural limit on all predictive AI that no increase in computational power can dissolve.
There is a class of prediction that contains within it the seed of its own falsification. A forecast about a bank's solvency, once public, can trigger the run that makes the bank insolvent. A prediction that a given route will be congested can empty it. A model that identifies which neighborhoods will see crime, and directs policing accordingly, generates more recorded crime in those neighborhoods, confirming its prediction by producing what it forecast. What makes these failures distinctive is that they are not errors of calculation but consequences of the prediction's own influence: the predicted thing is an intelligent agent that can learn the prediction and respond. Oskar Morgenstern identified this structure in the 1930s, in his analysis of the impossible regress of perfect economic foresight, and he formalized it as the foundational problem that motivated game theory itself. The insight is now one of the most important and least appreciated constraints on what
Keep reading with YOU ON AI
Unlock the full book, 10,000+ field-guide entries, and a 1000+ thinker library. If you have a book code, register now — it takes a minute.