CONCEPT
Solomonoff Induction
The provably optimal method for predicting the future from the past—derived by folding algorithmic probability into Bayes’ rule and running forward—and the uncomputable ideal against which every real learning system, including today’s language models, can be precisely measured and found permanently short.
Solomonoff induction is the answer to the oldest problem in the philosophy of science, stated in the most rigorous way yet achieved. The problem is induction: we reason constantly from the observed to the unobserved, and there is no logical guarantee that the future will resemble the past. David Hume diagnosed this as an open wound in the eighteenth century.
Ray Solomonoff proposed a closure. His method: consider every computer program that could have generated the data you have observed so far; weight each program’s vote on the future by the inverse of its length, so that shorter programs count for more; blend the weighted predictions. The result is a predictor that begins by assuming the world is as simple as possible—Occam’s razor welded into the prior, operating automatically—and updates only as far as the evidence forces it to. By a precise mathematical argument, this predictor is
optimal: it converges on the truth