CONCEPT
Temporal-Difference Learning
The method by which an agent learns to predict the future not by waiting for outcomes but by comparing successive predictions—bootstrapping wisdom from the gap between what it expected a moment ago and what it expects now.
Temporal-difference learning is one of the foundational ideas in the science of how minds learn. Introduced by
Richard Sutton in its mature form in a 1988 paper, it solves a practical problem with a philosophical implication: an agent that must act and improve in a world does not get to wait for the final score. Instead of learning only when an episode ends, a TD learner adjusts its predictions continuously, comparing what it expected at each moment to what it expects a moment later, and closing the gap. The method is
bootstrapped—the agent uses its own later, slightly better-informed estimates as targets for its earlier ones, pulling itself forward by its own predictions. What makes this more than an engineering trick is its uncanny convergence with biology: in the 1990s, neuroscientists studying the dopamine system found that certain neurons compute exactly the quantity at the heart of TD learning—the difference between expected and received reward—suggesting that this is