CONCEPT
Distribution Shift
The condition in which the statistical properties of the data a machine learning system faces in deployment differ from those it encountered in training—the most common silent cause of AI failure in the world, and, in de Finetti's terms, the precise mathematical name for exchangeability breaking down.
Distribution shift is what happens when the world moves. A model trained on one slice of reality—one time period, one population, one context—is deployed against a different slice, and the statistical regularities it learned no longer hold. The failure can be gradual and invisible, or sudden and catastrophic: a fraud-detection model trained on yesterday's fraud meets the novel fraud invented precisely to evade it; a medical AI trained on data from one demographic population is deployed against another; a language model trained before a significant event is asked to reason about the world after it. In each case the model behaves as if the past and the future were interchangeable draws from the same urn, and they are not.
Bruno de Finetti's framework gives distribution shift its deepest name: it is the failure of
exchangeability, the collapse of the symmetry of belief that licensed learning from data