CONCEPT
Calibration as Trainable Skill
The empirical finding that the correspondence between confidence and accuracy improves through practice — assigning probabilities, tracking outcomes, adjusting based on feedback — a skill AI threatens to atrophy.
Calibration is the alignment
between stated confidence and actual accuracy: a well-calibrated forecaster who says 'seventy percent probable' is right seventy percent of the time.
Tetlock's research demonstrated that calibration is not an innate trait but a trainable skill that improves through
deliberate practice and degrades without it. The
Good Judgment Project showed that an hour of training in probabilistic reasoning produced measurable, durable gains in calibration, and that forecasters who practiced continuously — making predictions, scoring outcomes, adjusting confidence levels — maintained their advantage across years. Calibration is simultaneously the most important cognitive skill in an age of abundant information and the skill most threatened by AI tools that present all output with identical confident fluency.
In The You On AI Field Guide
The training protocol was deceptively simple. Forecasters were taught to think in granular probabilities rather than verbal estimates ('likely' versus 'sixty-five percent'), to consider base rates before case-specific details, to break complex questions into simpler components, and to update beliefs incrementally