Probability as Counting
In the [YOU] on AI Field Guide
The cycle’s central argument about AI capability and AI limitation runs through this concept. A large language model learns the statistical structure of human language with extraordinary precision, mapping the low-entropy regions of the space of all possible text—the configurations that constitute coherent sentences, plausible arguments, grammatical prose. The learning works because probability as counting is the right framework for this task: there are far more ways to arrange tokens into nonsense than into sense, and the model learns to favor the sense-shaped arrangements. This is the engine of the fluency that has made these systems transformative.
The limitation is the same as the power: the method captures which configurations are probable and is constitutively silent on which are true, meaningful, or right. The most probable sentence is not the wisest; the most likely continuation is not the most honest. Judea Pearl’s analysis converges here from a different angle: statistical patterns occupy only the first rung of the ladder of causation, and no accumulation of counting can climb to intervention or counterfactual reasoning. Boltzmann’s framework makes this point with maximum precision: the statistical method succeeds by averaging over particulars, and what gets averaged away is exactly the specific, determinate significance of any particular arrangement—which is, on current understanding, where meaning lives.
Origin
The insight emerged from Boltzmann’s effort in the late 1860s and 1870s to explain why thermodynamic laws hold. His formula, carved on his grave as S = k log W, equates entropy S to the Boltzmann constant k times the logarithm of W, the number of microscopic configurations consistent with the observed state. The formula was actually written in this compact form by Max Planck, who named the constant for Boltzmann in tribute. The insight it encodes is that macroscopic regularity—temperature, pressure, entropy—is the averaged behavior of an enormous number of microscopic events, each governed by ordinary mechanics. The appearance of law is the appearance of overwhelming probability.
Geoffrey Hinton and Terrence Sejnowski imported this mathematics directly into machine learning when they designed the Boltzmann machine in 1985, naming it explicitly for the physicist whose distribution sat at its core. The 2024 Nobel Prize in Physics, awarded to Hinton and John Hopfield, cited Boltzmann by name in the Academy’s scientific background document, acknowledging that the lineage from statistical mechanics to machine learning was not metaphorical but foundational.
Key Ideas
Entropy as multiplicity. The single most consequential reframing: a state’s entropy is not a property of the state itself but a count of how many ways there are to be in that state. High entropy means many ways; low entropy means few. The universe drifts toward disorder not because disorder is more natural but because disorder is more populous—almost every arrangement of molecules is disordered, so a system wandering through configuration space will overwhelmingly likely find itself in a disordered state.
The Boltzmann distribution applied to AI. When engineers set the “temperature” of a language model, they are directly invoking this concept: at high temperature, the model samples nearly uniformly from its probability distribution, exploring rarely visited configurations; at low temperature, it concentrates on its most probable outputs. The mathematical form is identical to Boltzmann’s formula for the distribution of molecular energies in a gas.
The gap the count cannot cross. Boltzmann’s statistics characterize the probable with complete fidelity and are structurally silent on the significant. A gas molecule has no meaning; averaging over its individual specifics loses nothing. A human sentence is constituted by its specific meaning, by the particular intention of a particular speaker in a particular situation. The method that succeeds for molecules fails for sentences precisely at the point where meaning begins—not because the method is imperfect but because meaning is not a statistical property.
Debates & Critiques
The central debate is whether the gap between probability and meaning is permanent or merely the current frontier of scaling. The scaling optimist argues that a system trained on enough instances of human meaning-making must absorb the causal and semantic structure that underlies the statistical patterns, because the patterns are the footprints of the structure. Boltzmann’s own framework resists this: the statistical method works precisely by discarding individual instances, by treating as equivalent all the configurations that look the same from outside. To recover meaning from the statistics would require inverting a step that is, by construction, not invertible—the step that loses the particular in the aggregate. Claude Shannon’s information entropy is closely related: Shannon proved that the entropy of a source measures its average uncertainty per message, and he explicitly analogized his measure to Boltzmann’s. The two concepts are structurally parallel in their power and in their silence on significance.
Further Reading
- Ludwig Boltzmann, Lectures on Gas Theory (1896–1898; trans. S.G. Brush, University of California Press, 1964)
- Roger Penrose, The Road to Reality (Knopf, 2004) — Chapter 27 on the second law and the direction of time
- David Lindley, Boltzmann’s Atom (Free Press, 2001)
- Geoffrey Hinton et al., “A Learning Algorithm for Boltzmann Machines,” Cognitive Science 9 (1985)