CONCEPT
Brier Score
The quadratic scoring rule — measuring the squared distance between predicted probability and observed outcome — that Tetlock used to evaluate expert forecasts and that AI output urgently requires.
The Brier score, developed by meteorologist Glenn Brier in 1950, quantifies forecast accuracy by measuring the mean squared difference between predicted probabilities and actual outcomes. A perfect forecast receives a score of zero; the worst possible forecast receives a score of two. The score is 'proper,' meaning it incentivizes honest reporting of subjective probabilities rather than strategic hedging. Tetlock adopted the Brier score as the primary metric for Expert Political Judgment because it captured both calibration (whether seventy-percent predictions are right seventy percent of the time) and resolution (whether the forecaster distinguishes between more- and less-probable events). The Brier score made expert accountability possible by transforming vague predictions into measurable claims.
In The You On AI Field Guide
The mathematical simplicity of the Brier score conceals its conceptual power. For a binary event (will it happen or not?), the score is calculated as: BS = (f - o)², where f is the forecast probability and o is the outcome (1 if the event occurred, 0 if