The Convex Hull of Knowledge
The convex hull of a set of points is the smallest convex shape containing all of them — the rubber band stretched around a set of pins. Every point inside can be reached by weighted combinations of the pins; every point outside cannot. Applied to large language models, the training corpus defines a set of points in a very high-dimensional space of possible texts. The convex hull is the set of all texts reachable by combining patterns from the training data. The model's outputs, governed by the statistical regularities it has learned, lie within this hull or in its immediate neighborhood. A point outside the hull would require variation that the model's optimization explicitly penalizes. This is the geometric structure behind Campbell's claim that AI's variation is always directed — and the frame that reveals why genuine discovery requires something the technology's architecture structurally prevents.
In the AI Story
The convex hull frame clarifies what surprising-to-the-user means versus genuinely novel. A user whose knowledge is narrower than the model's training data will regularly encounter AI outputs that surprise her — connections she had not made, domains she had not integrated, syntheses she had not performed. These surprises feel like discovery. They are, in Campbell's framework, retrievals. The connection was always within the convex hull. The user simply could not see the relevant region from where she stood.
Genuine novelty requires something more: a configuration outside the convex hull, a point in the possibility space that no existing pattern predicts. Fleming's penicillin, Röntgen's X-rays, Goodyear's vulcanization — all were points outside the hull of their era's knowledge. They could not have been reached by refinement of existing patterns, only by blind probes that no directed search would have generated.
The frame connects to Stuart Kauffman's concept of the adjacent possible — the set of configurations reachable from the current state by a single step. The convex hull is, approximately, the AI's adjacent possible: the region of configurations one step from the known that the training data's statistical structure supports. Kauffman observed that transformative discoveries often require multi-step paths through regions the current state does not predict. Next-token prediction is optimized for adjacency, not for the multi-step probe.
The frame has direct implications for the interpolation trap. Every AI output within the hull carries the surface features of novelty — specific combinations not produced before — while remaining structurally governed by the hull's boundary. The evaluator who cannot distinguish interior points from boundary points will systematically overvalue interpolation and undervalue the blind probes that alone can leave the hull.
Origin
The convex hull is a standard concept in computational geometry, applied to machine learning through the statistical learning theory of Vapnik and Chervonenkis. Campbell himself did not use the term, but the geometric intuition maps exactly onto his distinction between directed variation (interior search) and blind variation (boundary-crossing).
The application to AI outputs has been developed by researchers studying out-of-distribution generalization and the limits of next-token prediction. The Wolpert-Macready No Free Lunch Theorems provide a formal foundation: no algorithm outperforms random search across all problem landscapes, which is another way of saying directed search cannot reach points outside the space its assumptions define.
Key Ideas
The training corpus defines a geometric region. Not a list of texts but a shape in very high-dimensional space, whose interior is reachable by weighted combinations of the corpus's patterns.
AI outputs lie within the hull or at its boundary. Next-token prediction penalizes deviation from the statistical structure of the training data, making interior and near-boundary outputs probable and exterior outputs structurally rare.
Surprise-to-the-user is not departure-from-the-hull. A user's surprise measures the user's knowledge relative to the model's. It does not measure the output's genuine novelty relative to the aggregate of human knowledge.
The most transformative discoveries lie outside the hull. The history of penicillin, X-rays, and continental drift suggests that the configurations producing paradigm shifts are, by definition, in regions that existing patterns do not predict.
Debates & Critiques
The hardest question is empirical: can language models produce outputs outside the convex hull? Proponents of emergent capabilities argue that scale produces genuine extrapolation, citing examples of abilities that appeared at threshold model sizes. Skeptics argue that these emergences are better explained as unlocking of capabilities that were implicitly present in the training data but required scale to express. Campbell's framework does not resolve this, but insists that the answer matters more than the discourse acknowledges.
Appears in the Orange Pill Cycle
Further reading
- Wolpert, D. H., & Macready, W. G. (1997). No Free Lunch Theorems for Optimization.
- Kauffman, S. (2000). Investigations.
- Simonton, D. K. (2022). The Blind-Variation and Selective-Retention Theory of Creativity.
- Bender, E. M., et al. (2021). On the Dangers of Stochastic Parrots.
- Wei, J., et al. (2022). Emergent Abilities of Large Language Models.