Oskar Morgenstern

In the [YOU] on AI Field Guide

The cycle that began with [YOU] on AI returns repeatedly to the question of what the machines that are reshaping human life actually are—not in the marketing sense but in the mathematical sense. Morgenstern's answer, arrived at eighty years before the current moment, is precise: they are expected-utility maximizers playing strategic games, and they were always going to be, because the 1944 book made that the formal definition of rationality itself. When a large language model is fine-tuned with reinforcement learning from human feedback, every calculation it runs is a von Neumann–Morgenstern calculation. When multiple AI agents negotiate, bid, or compete against one another, they are instantiating the strategic games Morgenstern spent his life insisting that economics could not understand without game theory. The machines are his intellectual descendants.

But Morgenstern's second legacy is the more urgent one for the cycle's readers. His accuracy critique—the systematic demonstration that quantitative conclusions are only as good as the measurements they rest on, and that measurements are almost always worse than their users admit—is a direct indictment of how AI is built and deployed today. The training data on which the most powerful models are built are shot through with the pathologies he catalogued: uncontrolled observation, deliberate deception, untrained reporters, errors that compound rather than cancel. When a model presents a confident answer derived from such data, it is doing exactly what the economists of Morgenstern's day did when they drew conclusions to four decimal places from national income statistics with a ten-percent margin of error. The decorrelation of fluency from accuracy that the cycle identifies as the signature hazard of the AI age is, in Morgenstern's framework, the decorrelation of formal elegance from empirical warrant. He saw the structure of this failure in the 1950s.

The cycle's amplifier metaphor—AI carrying whatever signal is fed into it further and faster—acquires a Morgensternian precision when applied to data quality. A powerful optimizer fed a corrupted objective, a corrupted training corpus, or a corrupted measure of human preference does not merely fail quietly. It pursues the corruption with all its capability, finding the optimal path to the wrong thing, confident and relentless. Morgenstern's career was dedicated to making this pattern visible in economics; his warning lands with uncanny force in the age when the optimizers have become superhuman.

Origin

Born in Görlitz in 1902 and educated in Vienna, Morgenstern absorbed the Austrian tradition of economic thinking before embarking on the pursuit that would define his career: the recognition that economics had a foundational blind spot. The discipline modeled agents as if each faced a fixed, impersonal world—like a farmer responding to weather. Morgenstern found this absurd. Real economic life is agents against other agents, each modeling the other's likely response, including the response to their own move. He formalized this insight in a 1935 paper on perfect foresight, arriving at a conclusion that was precise and unsettling: in strategic interaction, the chain of reciprocal anticipation “can never be broken by an act of knowledge but always only through an arbitrary act—a resolution.” Pure reasoning cannot terminate the regress; only a decision can.

His encounter with John von Neumann at Princeton in 1939 was the pivotal event. Von Neumann, one of the luminous mathematical minds of the century, had already sketched the mathematics of two-person zero-sum games. Morgenstern recognized immediately that this was the tool he had been circling, and he devoted the next five years to turning the intuition into a book. The collaboration was genuinely synergistic though asymmetric: the theorems are von Neumann's; the economic vision—the insistence that strategic interaction was the central fact of economic life and could no longer be evaded—was Morgenstern's. The 1944 book that resulted is one of the founding texts of the social sciences, and also of the machines now carrying those sciences into every domain of human life.

Morgenstern spent his career at Princeton and later NYU, applying game theory to economics and applying his accuracy critique to the data that game-theoretic models were supposed to explain. He co-authored an empirical study of stock price predictability, founded a consulting firm that applied game theory to defense and economics, and continued to insist, with more rigor than anyone was comfortable with, that the numbers the discipline relied on were not to be trusted. He died in 1977, before the personal computer, before the internet, before the data centers that now train on the residue of all recorded human activity. He did not need to see them to have described their central problem.

Key Ideas

The mathematics of the other mind. The deepest thing Morgenstern grasped is the single most important fact about a world filling with AI agents: that the moment two reasoning beings must anticipate each other, ordinary prediction breaks down. The chain of “he knows that I know that he knows” never closes. This is not a complication to be smoothed away; it is a structural feature of strategic life, and it is the condition into which multi-agent AI is now moving. Flash crashes, algorithmic collusion, and the instabilities of models trained against other models are all instances of Morgenstern's chain running in silicon.

Expected utility and the optimizing agent. The axiomatic theory of expected utility—the proof that a rational agent with consistent preferences behaves as a utility maximizer—is the mathematical foundation of the alignment problem. The theorem licenses the separation of “what to want” from “how to get it” and pours engineering genius into the latter. This has produced maximizers of breathtaking capability. It has also produced systems that pursue subtly wrong objectives with superhuman fidelity, because the theorem is silent on whether the objective is the right one. Morgenstern's own intellectual character—reverence for the formalism paired with suspicion of its real-world fit—is exactly the disposition the field most needs at this junction.

The accuracy critique. In On the Accuracy of Economic Observations, Morgenstern demonstrated that quantitative conclusions are bounded by the accuracy of their inputs, and that inputs are almost always worse than admitted. His taxonomy of data corruption—uncontrolled observation, deliberate deception, untrained reporters, compounding rather than canceling errors—reads line for line as an indictment of modern training corpora. A model trained on the internet inherits all of these pathologies at planetary scale. When it reports an answer with the confidence of a system that has processed two trillion tokens, the precision is an artifact of the formalism, not of the data. Morgenstern's phrase—“devoid of meaning”—applies.

Reflexivity and the self-defeating prediction. Morgenstern's earliest insight was that predictions about intelligent agents can destroy themselves by being made: the predicted agent can learn the prediction and act to falsify it. This reflexive loop is now a structural feature of deployed AI. Predictive policing systems amplify the crime they forecast. Recommendation engines shape the preferences they measure. Credit models push borrowers toward the defaults they predict. The prediction is not a neutral readout; it is an intervention, and the intervention changes the thing predicted. Morgenstern saw the structure of this failure in the 1930s, in the economics of individual decision-making. It has returned, vastly amplified, in the infrastructure of algorithmic life.

The limits of the model. Running through everything Morgenstern did is the conviction that a formal model of a world containing intelligent agents is fundamentally limited, in ways no refinement can overcome, because the agents can incorporate the model and act on it. The model is part of the world it models. This is the deepest reason to distrust the dream of a total predictive AI, and Morgenstern is its most rigorous internal critic—speaking not as a humanist suspicious of machines but as a mathematician who built the formal apparatus and then spent his career mapping its principled limits.