The Representativeness Heuristic
The representativeness heuristic is the cognitive operation by which probability is judged through similarity to a prototype rather than through base-rate statistics. Asked whether a quiet, detail-oriented individual is more likely to be a librarian or a farmer, subjects choose librarian — because the description matches the stereotype, despite farmers vastly outnumbering librarians. The heuristic is efficient and usually adequate, but produces systematic errors when prototype-match and actual-frequency diverge. In the AI era, the heuristic is exploited by large language models whose output matches the surface features of good analysis — fluent prose, structured reasoning, appropriate hedging — regardless of whether the underlying claims are accurate. The output looks like correctness, and the representativeness heuristic reads this resemblance as evidence of correctness.
In the AI Story
The 1972 and 1973 papers by Kahneman and Tversky established representativeness as a foundational shortcut, demonstrating through experiments on the conjunction fallacy, base rate neglect, and insensitivity to sample size that subjects systematically substituted resemblance judgments for probability judgments. The classic Linda problem — in which subjects judged "Linda is a feminist bank teller" more probable than "Linda is a bank teller" because the former matched the prototype — became the canonical demonstration that representativeness can override elementary logic.
In the AI context, representativeness operates on both the human side and the system side. On the human side, the evaluator judges AI output by how much it resembles good human output — its fluency, structure, confidence, and references — rather than by independent verification of its claims. On the system side, LLMs are trained to produce outputs that are representative of the patterns in their training data, which means they optimize for prototype-matching at the level of form, not for accuracy at the level of content.
The Orange Pill's account of the Deleuze incident — in which Claude produced a passage connecting flow state to a concept wrongly attributed to Deleuze — is a clean illustration. The passage was representative of good philosophical analysis in every surface feature. It was accurate in no important sense. The representativeness heuristic reads such output as reliable because it matches the prototype of reliability.
The representativeness heuristic interacts with the smoothness-overconfidence loop: the uniformly polished surface of AI output removes the effort-and-struggle cues that normally calibrate confidence, and the representativeness match supplies a false substitute for verification. The evaluator feels she has verified when she has merely pattern-matched, and the feeling is indistinguishable from actual verification from inside the experience.
Origin
Representativeness was formalized in the 1972 paper 'Subjective Probability: A Judgment of Representativeness' by Kahneman and Tversky. The 1973 and 1974 papers extended the framework and introduced the conjunction fallacy as a stark demonstration.
The implications for AI evaluation were not explored in the original program — LLMs did not exist — but the framework applies with force. The Turing test, in retrospect, can be read as a test of representativeness: a system passes by producing outputs that are representative of human cognition, not by instantiating human cognition itself.
Key Ideas
Prototype substitution. The cognitive system substitutes the question "how much does this resemble the prototype?" for the harder question "how probable is this?"
Surface for substance. Fluency, structure, and confidence are read as evidence of accuracy because they are features of accurate output in the historical record of human cognition.
Base rate neglect. Representativeness judgments systematically ignore statistical base rates, producing the conjunction fallacy and related errors.
AI exploitation. Systems that optimize for surface features of good output exploit the representativeness heuristic in human evaluators, who read resemblance as evidence of reliability.
Self-verifying error. The evaluator using representativeness does not experience herself as pattern-matching; she experiences herself as evaluating, which makes the error invisible from inside.
Debates & Critiques
Whether representativeness is a single heuristic or a family of related processes remains debated. The competing fast-and-frugal tradition argues that representativeness-like processing is often ecologically rational in natural environments, and that the systematic errors Tversky documented reflect laboratory artifice rather than everyday performance. Defenders respond that the modern information environment resembles the laboratory more than the savanna.
Appears in the Orange Pill Cycle
Further reading
- Kahneman, Daniel and Amos Tversky, 'Subjective Probability: A Judgment of Representativeness' (Cognitive Psychology, 1972)
- Kahneman, Daniel and Amos Tversky, 'On the Psychology of Prediction' (Psychological Review, 1973)
- Tversky, Amos and Daniel Kahneman, 'Extensional versus Intuitive Reasoning: The Conjunction Fallacy in Probability Judgment' (Psychological Review, 1983)
- Gilovich, Thomas, Dale Griffin, and Daniel Kahneman, eds., Heuristics and Biases: The Psychology of Intuitive Judgment (Cambridge University Press, 2002)