The Bellman Equation

The Bellman equation is the single most important equation in the theory of sequential decision-making, and it is the direct ancestor of reinforcement learning as a discipline. Richard Bellman wrote it at the RAND Corporation in the early 1950s as the defining property of a value function: the value of being in a state is the maximum, over available actions, of the immediate reward plus the discounted value of the state that action leads to. This recursive self-reference is what makes it powerful: to know the value of the present, you assume the future is already optimally handled, which means solving the hard global problem of choosing a whole sequence of actions reduces to a local problem of choosing one action, repeated. Modern deep learning agents—from game-playing engines to fine-tuned language models shaped by reinforcement learning from human feedback—are, at bottom, systems that learn to approximate the value function this equation