Exploratory Data Analysis

In the [YOU] on AI Field Guide

The cycle built around [YOU] on AI places human judgment at the center of the AI moment—the capacity to look, to doubt, to ask whether the question was right. EDA is Tukey's name for that capacity applied to data. Its absence from modern AI practice is not an oversight but a structural consequence of scale: training corpora are simply too large for any human examination to be exhaustive. But the alternative to looking is blindness, and blindness, Tukey insisted, is where the most precise and most wrong answers are born. The emerging discipline of data-centric AI—the turn from model improvement to data improvement, from benchmark optimization to dataset documentation, from aggregate metrics to per-subgroup performance audits—is the spirit of EDA translated into practices that can survive at scale.

The connection to AI safety is direct. A model trained on unexamined data inherits the biases, errors, and selection effects of that data with perfect fidelity. The famous cases—facial recognition systems that fail on faces underrepresented in their training sets, medical models that work for the populations they were trained on and fail for those they were not, language models that reproduce the perspectives dominant in their corpus and erase the rest—are all, at root, failures of examination. They are the failures of a pipeline in which no one stopped to ask what the data contained, whose data it was, and what the gaps would do to the conclusions. Tukey would have recognized these failures instantly. He built a discipline to prevent them.

Origin

Tukey began developing the ideas of EDA in lectures and papers through the 1960s, crystallizing them in his landmark 1962 paper “The Future of Data Analysis.” That paper called for a new discipline, distinct from mathematical statistics, organized around the actual practice of learning from numbers rather than the theoretical analysis of inference procedures. EDA was partly a response to the gap Tukey saw between what statisticians taught and what analysts actually needed: the ability to approach data that might tell you something unexpected, without presupposing what it would say.

The 1977 book brought the ideas to a wide audience with deliberately low-tech tools: hand-drawn diagrams, work that could be done with pencil and graph paper. The aesthetic was intentional. Tukey wanted methods that a human eye could deploy directly, without the mediation of computation, because the eye is what catches the unexpected. He built the stem-and-leaf display so the shape of a distribution could be read off the raw numbers; the box plot so the median, spread, and outliers could be seen at a glance; the two-way table so interaction effects would become visible before any significance test was run. The tools were instruments for a faculty he trusted: the human capacity to see pattern and anomaly.

Key Ideas

Surprise-readiness. The defining attitude of EDA is openness to being wrong about what the data will show. Where confirmatory analysis tests a specific hypothesis, EDA approaches the data with no fixed expectation, alert to whatever structure or anomaly emerges. This is the detective's stance rather than the prosecutor's: the goal is discovery, not confirmation. It is exactly the stance that large-scale, data-blind training abandons, because a training objective is precisely a fixed expectation—minimize this loss function—with no mechanism for the unexpected to surface.

Resistant summaries. Tukey built EDA around statistics that are robust to extreme values: the median rather than the mean, the interquartile range rather than the standard deviation, trimmed and Winsorized estimators rather than ordinary ones. Resistance was an ethical as much as a technical principle: a summary that lets one bad value dominate the conclusion is not an honest description of the data. Modern AI training procedures are generally not resistant—squared-error objectives give quadratically growing weight to outliers—and the consequence is exactly what Tukey would have predicted: a handful of corrupted or mislabeled examples can systematically warp what a model learns.

The outlier as message. Tukey did not treat outliers as noise to be discarded. He treated them as signals to be examined: they might be errors, in which case you want to catch them, or they might be the most interesting things in the dataset—the anomaly that breaks an assumption and teaches something new. EDA's explicit flagging of outliers embodies this dual respect: see them, do not let them distort the summary, and then go look at them. The AI equivalent is out-of-distribution detection, the attempt to recognize when a model is being asked to operate beyond the range of its training. The failure of models to make this recognition reliably—to extrapolate confidently into regions where they have no real support—is a failure of Tukey's outlier logic applied to inputs.

The modern descendants of EDA. The literal box plot cannot be applied to a trillion-token training corpus. But Tukey's question—what is the right way to see your data when exhaustive examination is impossible?—has generated a set of modern practices that carry his spirit forward. Datasheets for datasets (documentation of provenance, composition, and limitations), model cards (performance breakdowns by subgroup and use case), embedding visualizations (dimensionality reduction that lets human eyes see structure in high-dimensional spaces), automated bias audits: each is an attempt to recover, at scale, the epistemic humility Tukey's tools made possible at human scale. Human-AI collaboration in data analysis may be the most faithful modern form of EDA: the machine does the exhaustive search the human cannot, while the human does the judgment the machine cannot.