Questioning as the Core of Judgment
Tetlock's forty-year research program is, at its deepest level, about the quality of questions. The superforecaster does not begin with an answer; the superforecaster begins with the question: what am I actually trying to predict? The question forces decomposition of vague predictions into specific, scoreable claims. It forces consideration of the reference class (outside view), the distinctive features of the case (inside view), and the confidence level warranted by the evidence. In the AI age, the capacity to ask such questions becomes the primary human contribution to human-AI collaboration. The machine produces answers with extraordinary fluency. The human who cannot formulate the question that would test whether the answer is correct has outsourced the only cognitive operation that justifies keeping the human in the loop.
In the AI Story
The structure of a good forecasting question is specific and demanding. It must be falsifiable: there must be an observable outcome that would prove it wrong. It must be time-bound: 'eventually' is not a timeline. It must be probabilistic: assigning a number forces the forecaster to calibrate confidence rather than gesturing at it verbally. And it must be decomposable: complex questions should be broken into simpler components that can be evaluated separately and then aggregated. The discipline of formulating questions that meet these criteria is itself a filter on overconfidence, because vague questions conceal the uncertainty that specific questions reveal.
Segal's twelve-year-old asking 'What am I for?' is, in Tetlock's framework, a paradigmatic good question — open-ended, genuine, and resistant to the kind of answer that an AI system excels at producing. The question is not asking for information ('what jobs will exist in 2040?') but for meaning ('what is the purpose of human effort when machines can do what I do?'). Meaning-questions do not have answers that can be looked up, calculated, or generated by text prediction. They have responses that require the questioner's sustained engagement with the discomfort of not-knowing. The AI can provide eloquent responses. The eloquence is orthogonal to whether the response actually addresses what the child needs, which is not an answer but the experience of being taken seriously by an adult who will sit with the question's weight.
The questioning muscle, as Segal calls it, is the capacity to notice when an answer is too smooth, too complete, too confident for the evidence to support. The muscle develops through practice against resistance: asking questions, receiving answers that do not satisfy, asking better questions, iterating. The AI environment threatens the muscle by eliminating the resistance. The answer arrives immediately, comprehensively, polished. The questioning cycle that would build the capacity to question well is short-circuited by the availability of an answer that is good enough to stop the questioning. The child who asks 'why?' and receives an AI-generated explanation learns to accept the explanation rather than learning to push past it. The muscle never develops, because the resistance that would build it has been smoothed away.
Origin
The primacy of questioning in Tetlock's framework builds on the Socratic tradition and on C.S. Peirce's pragmatism, both of which treat the formulation of a question as the most difficult and most important cognitive operation. Peirce's 1877 essay 'The Fixation of Belief' argued that genuine inquiry begins not with doubt-for-its-own-sake but with a specific, irritating uncertainty that demands resolution. The question must be real — must emerge from actual ignorance rather than rhetorical positioning. Tetlock's contribution was to operationalize this insight: the good forecasting question is real, specific, and scoreable, and the discipline of asking such questions is what separates calibrated judgment from confident performance.
Key Ideas
Question quality determines answer quality. A vague question produces a vague answer; a specific, decomposed, falsifiable question produces a forecast that can be scored and improved.
Decomposition reveals uncertainty. Breaking a complex question into components forces recognition of how much is actually unknown — the aggregation of component uncertainties exceeds the felt uncertainty of the complex question.
Falsifiability as discipline. A question that cannot be answered wrongly is not a genuine question but a prompt for narrative generation — the distinction AI's fluency makes urgent.
Meaning-questions resist AI. Questions about purpose, significance, and what is worth doing cannot be answered by text prediction, because the answer depends on the questioner's values and commitments.
Questioning atrophy. When answers arrive instantly and comprehensively, the iterative questioning that builds the capacity to question well is eliminated — the muscle atrophies through disuse.
Appears in the Orange Pill Cycle
Further reading
- Peirce, C.S. (1877). 'The Fixation of Belief.' Popular Science Monthly, 12, 1–15.
- Tetlock, P.E., & Gardner, D. (2015). Superforecasting, Chapter 4: 'Superforecasters.'
- Gadamer, H.-G. (1960). Truth and Method, Part II: 'The Logic of Question and Answer.'