CONCEPT
Algorithmic Probability
The probability of an output measured by the likelihood that a randomly generated computer program produces it—Ray Solomonoff’s formalization of Occam’s razor as a number and the simplicity bias that hides, usually unacknowledged, inside every successful machine learning system.
Algorithmic probability is the move that changed everything:
Ray Solomonoff made simplicity measurable. For two thousand years, Occam’s razor was a maxim, a gesture toward elegance that no one could operationalize. Tell two scientists to prefer the simpler theory and they will argue forever about which theory is simpler, because “simple” was a feeling, not a quantity. Solomonoff’s algorithmic probability ended that argument by giving simplicity a unit: the length of the shortest computer program that produces an object. Fix a universal computer; generate a random program by flipping a coin for each instruction; the algorithmic probability of a particular output is the probability that this random process produces it. Because short programs are vastly more likely to arise by chance than long ones—a ten-bit program is exponentially more probable than a thousand-bit program—outputs with short descriptions automatically receive high probability, and outputs that can only be produced by long, intricate programs receive low probability. Occam’s razor