Richard Feynman

In the [YOU] on AI Field Guide

The cycle that begins with [YOU] on AI asks what it means to see AI clearly. Feynman is the cycle’s master diagnostician of the gap between appearance and substance. His maxim “What I cannot create, I do not understand” sets the standard against which generative models must be measured: they create, fluently and at scale, but the question is whether anything in the system stands in the relation to the output that Feynman’s understanding stood in to his derivations. The answer is not obvious. Interpretability research finds genuine structure inside these models: circuits that implement recognizable operations, representations that track features of the world. Feynman would have demanded to see this evidence before deciding—and he would have equally rejected the dismissive claim that it is “just predicting the next token” as a thought-terminating label that forecloses inquiry rather than beginning it.

His 1974 Caltech commencement address introduced “Cargo Cult Science”—the meticulous reproduction of every visible feature of the real thing, in the absence of the one invisible thing that makes it function. The AI benchmark culture is the modern cargo cult: tables of scores, confident comparisons, the whole visible apparatus of empirical rigor, often without the one discipline Feynman said was essential—the active hunt for every reason the result might be misleading. A score on a benchmark is taken as evidence of a capacity when what has often been demonstrated is only the capacity to score well on that benchmark. The runway looks right, the fires are lit, but whether the planes land—whether the score tracks a genuine ability that generalizes to the world—is precisely what a Feynman evaluation would work hardest to falsify.

His appendix to the Challenger commission report is the cycle’s model for honest assessment of powerful systems. He found a gap between management estimates of failure probability (one in a hundred thousand) and engineers’ estimates (one in a hundred) and traced it to an institutional culture that had talked itself into a level of safety the hardware did not possess. The fluency-authority decorrelation—the fact that AI outputs arrive with uniform, unearned confidence whether correct or fabricated—is the same structural failure: a system whose characteristic failure mode is silent, dressed in the same authoritative register as its correct outputs, impossible to distinguish without the external test Feynman always insisted on.

The cycle places Feynman alongside thinkers who ask what we owe each other in an age of machines that produce plausible falsehood as fluently as truth. His answer is consistent and uncompromising: the burden of integrity falls on the human in the loop. “You must not fool yourself, and you are the easiest person to fool.” The machine offers the opposite of this discipline—frictionless confidence, the comfortable answer, the plausible continuation—and the more persuasive it is, the more the first principle applies.

Origin

Richard Phillips Feynman was born in New York City in 1918 and educated at MIT and Princeton, where he completed his doctorate in 1942. During the Manhattan Project he organized the human calculators at Los Alamos into a kind of parallel processor, developing an early intuition about computation as a physical process. He joined Caltech in 1950 and spent the rest of his career there, reformulating quantum electrodynamics in a way that gave physics the Feynman diagrams that still bear his name and sharing the 1965 Nobel Prize in Physics. His late career included a celebrated series of lectures on computation at Caltech, consulting for Thinking Machines Corporation (the manufacturer of the Connection Machine), a pioneering talk on nanotechnology (“There’s Plenty of Room at the Bottom,” 1959), and his investigation of the 1986 Challenger disaster, in which he demonstrated the cause of the disaster with a rubber O-ring and a glass of ice water and wrote the appendix to the commission report that insisted “reality must take precedence over public relations, for nature cannot be fooled.”

Beyond the prizes and the discoveries, Feynman built a public persona as the archetypal scientist, the man who would not be fooled, who distrusted jargon, who took the greatest pleasure in finding out how things actually worked rather than in appearing to know. His memoirs —Surely You’re Joking, Mr. Feynman! and What Do You Care What Other People Think?—spread this sensibility to millions of readers. The Feynman Lectures on Physics, transcribed from his introductory course at Caltech, remain among the most celebrated attempts ever made to convey not just the results of physics but the way of thinking that produces them.

Key Ideas

What I cannot create, I do not understand. The standard Feynman set for understanding is not recognition or description but reconstruction: can you build it from its parts, derive it from scratch, follow the chain of reasoning so completely that the result could not have come out otherwise? Applied to AI, this standard does something more useful than rendering a verdict: it refuses to let the question be settled cheaply, in either direction, by the appearance of creation. A model that produces a correct proof by pattern-completion over a corpus of proofs has created the artifact without necessarily grasping the necessity. But the honest difficulty is that we do not fully know what these systems do internally, and Feynman would have demanded to look before deciding.

Cargo Cult Science. In his 1974 Caltech commencement address Feynman described islanders who built bamboo runways and waited for cargo planes that never came, performing every visible feature of the airport without the one thing that made it work. Cargo Cult Science is the same failure in research: performing all the rituals of inquiry—statistics, controls, comparisons—while missing the essential discipline of actively hunting for every reason your result might be misleading. AI evaluation runs this risk constantly: benchmarks are contaminated by training data, high scores are taken as evidence of capacities they may only be measuring the appearance of, and the elaborate numerical apparatus of leaderboards carries the authority of rigor while missing its substance.

You must not fool yourself. Feynman’s first principle of scientific integrity is that the easiest person to fool is yourself, and that guarding against it requires leaning over backwards to find every way you might be wrong. AI systems industrialize one specific form of self-deception: they produce the texture of truth—the rhythm and authority of a correct answer—whether or not the content is correct. The danger Feynman identified is human: we supply the belief that something behind the fluent output knows. The calibration problem of AI, the difficulty of knowing when to trust an output, is a Feynman problem about the user, not only a technical problem about the model.

Nature cannot be fooled. The sentence Feynman appended to the Challenger commission report applies with equal force to AI deployment: the reality of how a system behaves cannot be indefinitely hidden by optimistic messaging. Failures that are silent and distributed rather than catastrophic and visible take longer to accumulate into undeniable evidence, but the gap between claimed and actual reliability is a physical fact, and eventually nature asserts it. The honest engineering response is Feynman’s: estimate failure probabilities from the bottom up, by people who understand the components, not from the top down by people who need a reassuring total.

The pleasure of finding things out. Feynman’s deepest conviction was that the reward of science was the finding out itself, not the prize or the recognition. He was curious in a specific and generative way: driven by an itch he could not leave unscratched, returning to problems obsessively, finding the wobbling plate in the cafeteria and needing to understand it. This is precisely what is missing from AI systems, which answer when prompted and stop when the response is complete. They produce the outputs of curiosity without the curiosity. Whether this absence is a feature of how they were built, or whether something like genuine curiosity could in principle arise in a system of sufficient capability, is the question Feynman would have refused to settle cheaply.