Calibration is the alignment between stated confidence and actual accuracy: a well-calibrated forecaster who says 'seventy percent probable' is right seventy percent of the time. Tetlock's research demonstrated that calibration is not an innate trait but a trainable skill that improves through deliberate practice and degrades without it. The Good Judgment Project showed that an hour of training in probabilistic reasoning produced measurable, durable gains in calibration, and that forecasters who practiced continuously — making predictions, scoring outcomes, adjusting confidence levels — maintained their advantage across years. Calibration is simultaneously the most important cognitive skill in an age of abundant information and the skill most threatened by AI tools that present all output with identical confident fluency.
The training protocol was deceptively simple. Forecasters were taught to think in granular probabilities rather than verbal estimates ('likely' versus 'sixty-five percent'), to consider base rates before case-specific details, to break complex questions into simpler components, and to update beliefs incrementally as evidence accumulated. The training did not teach domain knowledge — it taught a method for relating confidence to evidence. Participants who absorbed the method became measurably more accurate, not because they knew more facts but because they knew how much weight their facts could bear. The improvement was specific and replicable across domains: geopolitical forecasting, medical diagnosis, engineering risk assessment, anywhere prediction under uncertainty was required.
Calibration degrades without practice. The Good Judgment Project documented that forecasters who stopped participating in tournaments — who stopped making scoreable predictions and receiving feedback — returned to baseline calibration within months. The skill is not a permanent acquisition but an ongoing practice. This finding has direct implications for AI-augmented work: professionals who accept AI output without the evaluative effort that would generate calibration data are not merely being careless in individual instances. They are allowing the skill to atrophy through disuse, the way a musician's technique degrades when they stop practicing scales. The degradation is invisible until the demand arrives — until a situation requires the full strength of the capacity, and the capacity is no longer there.
The AI age has created an environment structurally hostile to calibration maintenance. AI systems present outputs with uniform confidence regardless of underlying certainty. There are no vocal hesitations, no furrowed brows, no tonal shifts signaling doubt. The professional reviewing AI output must generate all evaluative effort internally, without environmental prompts. The tight feedback loops that the Good Judgment Project relied on — predictions scored within months, calibration curves updated in real time — are largely absent from professional AI use. The brief goes to the judge, the code goes to production, the strategy goes to implementation, and the feedback, when it arrives, is too delayed and too confounded by other variables to support the learning that calibration requires. Calibration atrophies not through malice but through environmental deprivation of the conditions it needs to survive.
The empirical study of calibration dates to the 1970s, when psychologists began documenting systematic overconfidence in expert judgment. Early work by Stuart Oskamp, Baruch Fischhoff, and Paul Slovic established that professionals in multiple domains — physicians, lawyers, engineers — assigned higher confidence to their judgments than accuracy warranted, and that feedback alone did not reliably improve calibration. Tetlock's contribution was to demonstrate that calibration could be improved through structured training combined with specific practice conditions: granular probability estimates, immediate feedback, and a culture that rewarded accuracy over confidence. The Good Judgment Project was the proof of concept, showing that ordinary people could be trained to calibrate their confidence to a degree that exceeded professional analysts who had never received such training.
Confidence-accuracy correspondence. A calibrated forecaster's stated confidence level matches their actual hit rate across predictions at that confidence level — seventy percent means right seventy percent of the time.
Training produces durable improvement. One hour of instruction in probabilistic reasoning measurably improves calibration, and the improvement persists as long as the forecaster continues practicing.
Feedback is essential. Calibration cannot improve without knowing whether predictions were correct — the scoreboard is the mechanism through which the skill develops and maintains itself.
Granular probabilities force honesty. The act of assigning a specific number rather than a verbal estimate ('likely,' 'probable') makes overconfidence visible and correctable.
Detraining is rapid. Forecasters who stop making scoreable predictions return to baseline (poor) calibration within months — the skill requires continuous exercise against resistance.