Counterfactual Reasoning
In the [YOU] on AI Field Guide
If [YOU] on AI asks what the human contribution is in a world where machines can answer any question, counterfactual reasoning is one of Pearl's most precise answers—and it converges, strikingly, on the book's own. The cycle locates the irreducibly human in the capacity to ask, to wonder, to care about a world that might have been otherwise; Pearl locates it on the third rung, in the mind that can grieve over the road not taken. Both are pointing at the same faculty from different directions: the creature with stakes in the world, that must choose how to spend finite time, is also the creature that can imagine the world it failed to make.
The rung gives rigorous structure to a claim the cycle makes about consciousness and the human difference. Pearl does not say counterfactual thought is magical; he says it is computational—a definite operation that could in principle run on a machine. What he denies is that we have built such a machine, or that the current trajectory of curve fitting leads there. The counterfactual animal is, for now, the only animal there is, and the gap between it and our most powerful machines is the gap between the third rung and the first.
This is why the rung bears so directly on the cycle's warnings about trust. The opacity of modern AI—its inability to explain its own outputs—is, in Pearl's framework, not an incidental flaw to be engineered away but a direct consequence of its position on the ladder. Explanation is a counterfactual act: to explain why something happened is to identify what would have had to be different for it not to have happened. A first-rung system cannot explain because explanation lives on the third rung.
Origin
The rung emerged from Pearl's effort to formalize the most human of questions within the same mathematics that gave him Bayesian networks and the do-operator. Counterfactuals had long been a battleground for philosophers—how can a statement about a world that did not happen have a truth value at all?—and Pearl's contribution was to make them calculable. Given a complete structural causal model, one can run the world backward to the moment of divergence, alter the chosen variable, and run it forward again down the path not taken, computing the difference. Counterfactuals are, in his hierarchy, the most information-hungry of all questions, demanding more of the model than either association or intervention.
Pearl traces the faculty deep into the human story. The ability to reason about alternatives to the actual, he argues, is what separated our ancestors from every other species and made science, law, and morality possible—a cognitive leap whose echo the cycle hears in its own account of the river of intelligence finding the channel of symbolic thought some seventy thousand years ago. To imagine the world as it is not is the precondition for changing it, for assigning responsibility, for asking what if.
The mathematics he built to make this rigorous is part of the achievement for which he won the 2011 Turing Award. It is also, he insists, the part of his framework most beyond current engineering—the destination he believes is reachable but whose road we have not yet found.
Key Ideas
The world that cannot exist. A counterfactual imagines an alternative contradicted by fact: the patient took the drug, yet we ask what would have followed had she not. Reasoning about it means holding the actual world and the impossible alternative at once—an operation that requires running a causal model backward and then forward along a different branch.
The foundation of moral life. Blame, regret, and responsibility each depend on a counterfactual. To say someone should have acted otherwise is to claim they could have—that a world existed in which they chose differently and the difference mattered. Strip away the counterfactual and the entire edifice of responsibility collapses. Ethics, on Pearl's view, is counterfactual reasoning applied to one's own conduct.
The engine of explanation. To explain why something happened is to identify the difference that made the difference: the fire occurred because of the spark, and we know this because, counterfactually, no spark, no fire. A system that cannot reason counterfactually cannot, in the deepest sense, explain anything—it can describe, predict, and pattern-match, but it cannot answer why in the way the question demands.
Why machines cannot reach it. A first-rung system has no model with which to construct the alternative world, and no model of itself as an agent whose actions might have been otherwise. It can be trained on human judgments of right and wrong and learn to produce the words a moral agent would produce, but this is mimicry of the surface. To call such a system moral, or immoral, is a category error—morality lives on the third rung, and the machines are on the first.
The reflexive counterfactual and the self. The deepest version is the one a mind turns upon itself: to ask what I could have done differently, to hold myself responsible, to regret. This requires a model not just of the world but of oneself as an agent whose choices could have been otherwise—a capacity Pearl links to selfhood and to something like free will, and one our pattern-matching machines, which are patterns all the way down with no represented agent at the center, entirely lack.
Debates & Critiques
Whether counterfactual capacity could ever be built is genuinely open, and Pearl leaves the door ajar. His own optimism—he has said plainly that we will one day have machines that reason about their own actions, perhaps possessing something like free will—rests on the claim that the third rung is a computational achievement, not a magic threshold. The disagreement runs in two directions. Stuart Russell builds the faculty into his design for safe machines: a system that reasons counterfactually about whether its planned action truly serves human preference is one that welcomes correction and the off switch, making the third rung a safety feature rather than a curiosity. Geoffrey Hinton, by contrast, suspects that the line between predicting language and understanding it is thinner than Pearl supposes—that a system trained to predict text well enough may already construct internal models rich enough to support something like counterfactual inference. Gary Marcus sides with Pearl: a network that has only learned correlations, he argues, cannot grasp the causal structure that counterfactuals require, no matter the scale. The rung remains the cleanest test case for the question of whether the human difference is reproducible.
Further Reading
- Judea Pearl & Dana Mackenzie, The Book of Why (Basic Books, 2018), esp. the chapters on counterfactuals and the third rung.
- Judea Pearl, Causality: Models, Reasoning, and Inference (Cambridge University Press, 2nd ed. 2009), ch. 7 on the logic of counterfactuals.
- David Lewis, Counterfactuals (Harvard University Press, 1973) — the philosophical backdrop Pearl formalized.
- Judea Pearl, “The Algorithmization of Counterfactuals,” Annals of Mathematics and Artificial Intelligence 61 (2011).
- Daniel Kahneman & Dale Miller, “Norm Theory: Comparing Reality to Its Alternatives,” Psychological Review 93 (1986) — the psychology of counterfactual thought.