CONCEPT
Probability as Counting
Boltzmann’s foundational reduction of thermodynamic law to combinatorics—entropy is the logarithm of the number of microscopic configurations that produce a given macroscopic state—and the engine beneath all of machine learning.
Probability as counting is the deepest idea
Ludwig Boltzmann gave to science, and it is the hidden foundation of every AI system operating today. Before Boltzmann,
entropy was a bookkeeping quantity in the theory of steam engines: a number that went up when heat spread out, defined by what it did rather than what it was. Boltzmann told us what it is. A state is low-entropy when very few microscopic arrangements produce it; a state is high-entropy when overwhelmingly many arrangements produce it. The gas spreads through the box not because any force pushes it toward disorder but because the spread-out configuration is backed by astronomically more microscopic arrangements than any ordered configuration could be. The second law of thermodynamics, apparently the most absolute law in nature, turns out to be a statement of astronomical likelihood rather than logical necessity. Boltzmann replaced a law with a probability so extreme it masquerades as one. The same logic governs
machine learning at its foundation: the space of all