CONCEPT
Distributional Semantics
The linguistic theory—summarized in J.R. Firth’s slogan that “you shall know a word by the company it keeps”—that meaning is substantially constituted by patterns of co-occurrence, and which became, scaled into neural networks, the conceptual engine of the AI transition.
Distributional semantics is the theory that the meaning of a word is encoded in the contexts in which it appears. You understand what
king means partly because of its proximity to
queen,
throne,
reign, and
crown—and its distance from
spatula or
photosynthesis. On this view, meaning is not, in the first instance, a matter of words pointing at things in the world; it is relational, constituted by a word’s position in a vast network of other words. The British linguist J.R. Firth expressed this in 1957 with a slogan that would prove more consequential than he could have imagined: “You shall know a word by the company it keeps.”
Christopher Manning took this slogan and turned it into engineering. If meaning lives in distribution, then a machine that measures those distributions at sufficient scale might learn meaning—which is precisely what the word vectors, attention mechanisms, and
large language models that Manning