CONCEPT
Intelligence as Compression
Ilya Sutskever's thesis that learning and compression are, at a deep level, the same activity—that to find the short description of the data is to recover the generative structure of the reality that produced it, and that this recovery is what understanding, prediction, and intelligence all amount to.
The intelligence-as-compression thesis holds that the learning achieved by a large neural network and the discovery of deep regularities in the world are not merely analogous but structurally identical: a model trained on a corpus vastly larger than its parameter count cannot memorize what it has seen, so it is forced to compress—to find the rules and regularities that generate the data rather than store the data itself. What it encodes in its weights is not the surface of the text but the generative structure beneath it, and a short description of a vast body of data is precisely what we mean by understanding it.
Ilya Sutskever drew this connection explicitly, framing the training of
large language models not as the accumulation of statistical regularities but as an analogue of the compression project at the heart of science itself: Newton compressed the motions of the planets and