The Counterfactual Animal

EDO SEGAL: Dr. Pearl, you locate what makes us human on the third rung — the counterfactual, the road not taken, the what would have happened had I done otherwise. And you say it's the rung furthest from any machine. But there's a problem in this room, because the man across the table spent his life arguing that this kind of imagining is just more habit, more association — vivid, but not different in kind. So I want to stage the collision directly. Tell us what the counterfactual is and why no curve-fitter can reach it. And, Mr. Hume — get ready to take it apart, because Dr. Pearl is about to claim the most human thing there is, and you have spent two and a half centuries deflating exactly such claims.

PEARL: The third rung asks the strangest question a mind can ask: not what happened, not what happens if I act, but what would have happened had I acted differently — a question about a world that not only does not exist but cannot, because the alternative it imagines is contradicted by what actually occurred. The patient took the drug and died; would she have lived had she not taken it? That world is foreclosed; she did take it. To answer, a mind must do something extraordinary — hold the actual world and the impossible alternative side by side and compute the difference between them. This requires a full causal model: you run the world backward to the moment of divergence, change the one thing, and run it forward down the road not taken. And this, I claim, is the engine of nearly everything we call human. It is the engine of regret — there is no regret without the imagined better choice. It is the engine of blame and responsibility — to say you should have acted otherwise is to claim you could have, that a world existed in which you chose differently and it mattered. Strip away the counterfactual and the entire edifice of morality collapses; there is no ought without the imagined alternative. And it is the engine of explanation — to explain why the fire happened is to identify what would have had to be different for it not to happen: no spark, no fire. A system that cannot reason counterfactually cannot, in the deepest sense, explain anything, regret anything, or be responsible for anything. It can describe, predict, pattern-match. It cannot say what if. And the machines are on the first rung, and what if is on the third, and the distance between them is the distance between a thing that records what happened and a creature that can grieve what did not.

EDO SEGAL: Mr. Hume. That's the most human thing in the room — regret, responsibility, grief, the road not taken. Take it apart.

HUME: I will, and I will do it with more discomfort than I have felt all evening, because Dr. Pearl has chosen his ground brilliantly — the counterfactual is indeed close to the heart of moral life, and any philosophy that cannot honor it is dead. So let me honor it and then dissolve it, which is my way. What happens when you imagine the patient surviving had she not taken the drug? You summon an idea — a faint copy of impressions you have actually had: patients, drugs, recoveries, deaths, all seen, all first-rung, in Dr. Pearl's terms. The imagination, which I showed combines and separates ideas with perfect freedom, assembles from these a scene that never occurred. But every element of that scene is a precipitate of past experience; the counterfactual is not a window into a foreclosed world, it is a recombination of remembered ones. And the feeling that the alternative is lawful — that she would have lived, not merely might have in some idle fancy — is, once again, nothing but the determination of a mind habituated by constant conjunction. You have seen un-drugged patients in like conditions recover, many times; custom carries you to expect it in the imagined case; and you call the carrying a computation over a causal model. I call it the imagination running on rails laid by habit. The counterfactual is the most sophisticated operation of association, Dr. Pearl — I grant it is sophisticated, and that the machines do it poorly. But sophisticated association is association. You have not found a third rung. You have found the imagination at the top of its powers, doing what it always does — copying impressions and recombining them — and you have mistaken its vividness for a different kind of knowing.

PEARL: No — and this is the hill, so let me defend it precisely, because Hume has made the strongest possible version of his case and it deserves the strongest possible answer. Hume says the counterfactual is recombined memory. Test it. The counterfactual "had she not taken the drug, she would have lived" makes a commitment that no recombination of observations can make: it commits to a specific value of a variable in a world that contradicts the data, and it can be wrong in a way that mere imagination cannot. If I have the causal model, I can compute the answer and you can check it against the model's structure; two people with the same model get the same counterfactual; it is not free fancy, it is determined — determined by the mechanisms, not by what I happen to remember. That determinacy is the proof it is not mere association: association gives you what is likely given what you saw; the counterfactual gives you what is necessary given the mechanisms, even for a case that violates everything you saw. Hume cannot get there, because his imagination, by his own account, is free — it can combine any ideas — and a faculty that can imagine anything cannot tell you what must have been. The counterfactual is not free. It is shackled to the model. That shackling is the third rung. And here is the test that settles it: a creature on the first rung, asked a counterfactual its memory never witnessed, must either refuse or confabulate. A creature with the model computes the unique answer. We do the latter, routinely, about cases no one ever saw — and the curve-fitter does the former, which is exactly why it hallucinates: it is a first-rung creature forced to answer third-rung questions, and with no model to shackle it, the imagination runs free, and free imagination, fluent and ungrounded, is precisely what we have learned to call a hallucination.

HUME: [long pause] That is the finest thing said against me tonight, and I am going to concede a real piece of it, because I did not earn my reputation by clinging when the argument turns. You are right that there is a difference between idle fancy and the disciplined counterfactual — that the latter feels constrained, that two careful reasoners converge on it, that it is not simply free. I underweighted that constraint; my account of the imagination as wholly free is too loose to capture the lawful counterfactual, and your hallucination point is a genuine wound — the machine's free-running fluency really does look like my unconstrained imagination, and the contrast with disciplined human counterfactual reasoning is real. Here is the piece I will not concede, and it is the deepest crack in your floor. You say the counterfactual is determined by the mechanisms. But the mechanisms — the model itself — where did it come from? From experience, structured by intervention, as we agreed in the matter of the child. So the determinacy you prize is inherited from past conjunctions, one level down. The counterfactual is shackled to the model, yes — and the model is shackled to custom. You have not escaped my circle; you have built, inside it, a magnificent machine that induces at the level of mechanism and then computes rigorously from that induced structure. The rigor is real. The computation is real. The third rung is real as an operation. But its foundation is the same habituated conjunction at the bottom, and so the necessity you feel in the counterfactual is, at the last, custom one storey down wearing the robe of necessity — exactly as the necessity in causation was. You have given me the most sophisticated thing the imagination can do. I have only ever asked where its materials come from. They come, every one, from impressions. And so I lose the operation and keep the foundation, which is the trade I have made all evening, and the one I will make to the end.

EDO SEGAL: [pause] Stop — because I think that's the truest exchange of the night, and the reader has to see its shape. Dr. Pearl proved the counterfactual is a real operation, different in kind, that the machine cannot do — and Mr. Hume conceded the operation and then said its foundation is still custom all the way down. They didn't cancel. They stacked. Pearl is right that there's a third rung the machine can't climb. Hume is right that the ladder is planted in the same soil as the bundle. The counterfactual is both — a genuine new operation, and a sophisticated flower of habit. [long pause] And it lands us exactly where my book lives: because if the counterfactual is the human move the machine can't make, then the candle in the darkness — the thing that doesn't get automated — is the question that begins with what if. The machine is an engine of what-was. You are an engine of what-might-have-been. We've reached the place where I hand the room to the two of you. The crossing. After this.