The Doom and the Category Error

**EDO SEGAL:** Pedro, of all your public positions the one that has cost you most is your rejection of the extinction story — the claim that a sufficiently advanced AI might pursue goals against our survival and, being more capable, prevail — the [existential-risk](https://www.youonai.ai/fieldguide/med/existential_risk) story. It's moved from the fringe to the center of elite discourse, and you call it a confusion dressed up as a danger. Make the case in full. And Karl — I want you to do something unusual before you respond. I want you to steelman the doom first. Tell us what the doomers get *right*, because you of all people know what it is to have watched a confident machinery of certainty end in catastrophe.

**DOMINGOS:** The doom rests on a category error, and once you see it you can't unsee it: it conflates intelligence with autonomy, and optimization with will. A learning system optimizes an objective that humans specify. It has no goals of its own, no drive to survive, no will to power — because those aren't properties of a mathematical optimization. They're properties of organisms shaped by billions of years of selection for survival and reproduction. The machine never went through that crucible. It does what its objective says and nothing else. The fear that it will spontaneously develop the ambition to dominate us mistakes the projection of human psychology onto mathematics for a discovery about mathematics. We're pattern-matchers that evolved to see agents everywhere — a face in the clouds, a will in the storm — and now we see a scheming mind in a gradient descent. The systems running credit and hiring and logistics aren't superintelligences plotting. They're brittle pattern-matchers deployed without oversight, and the harm is their stupidity at scale, not their cunning. Obsessing over the godlike future machine distracts from the stupid present one that's already ruining lives. That's the whole argument, and the institutional point on top: existential-risk alarmism functions, intended or not, as a lever to concentrate power — a rationale for [regulation](https://www.youonai.ai/fieldguide/med/ai_governance) that entrenches incumbents and a fundraising engine for the organizations describing the danger.

**POPPER:** Then let me steelman the doom honestly, because Pedro asked me to and because I have earned the right to fear runaway certainties more than most men alive. What the doomers get right is the asymmetry of catastrophe — that some errors are not survivable, and that for such errors the ordinary rhythm of conjecture, mistake, correction breaks down, because there is no one left to do the correcting. My whole method assumes we can learn from our refutations. It assumes the black swan kills the theory and not the theorist. The doomers are pointing at the one class of mistake where that assumption fails — where the refutation arrives in the form of extinction, and the lesson can never be applied. That is a genuine and serious limit on my own philosophy, and I will not wave it away. Where the experiment cannot be repeated, falsificationism offers thin comfort, and a precautionary logic must enter. So I give the doomers that much, and it is not nothing.

But — and here is where I come to Pedro's side, partway — what the doomers do *wrong* is the thing I spent my whole life fighting. They have built an unfalsifiable prophecy. Ask them what observation, between now and the end, would show the doom to be false, and you will find there is none — every reassuring fact is reinterpreted as the calm before the treacherous turn, every failure of the machine to destroy us is "not yet," every capability is evidence and every limitation is temporary. That is the exact structure of the historicist prophecies that justified so much killing in my century: a future so certain that any present sacrifice is warranted to prepare for it, and any doubt is naivety. A prophecy that cannot be refuted is not a scientific warning. It is a faith, and faiths that demand we reorganize society around an unprovable future are the oldest enemies of the open society there are.

**DOMINGOS:** That's exactly it, and you said the precautionary part more fairly than I usually do, so let me concede my half: I'm too quick to dismiss the unsurvivable-error class. There are mistakes you can't iterate out of, and for those, "wait and see" is not a method. I'll own that gap. But the doom as *practiced* isn't a careful precautionary argument about a narrow class of irreversible risks. It's a totalizing story that converts every fact into confirmation, and it's being used — whether the believers intend it or not — to argue that only a few trusted firms should be allowed to build, which is the most effective regulatory capture ever attempted. The unfalsifiable prophecy and the concentration of power are the same move. Karl spent his life showing that the people who claim certain knowledge of the future always end up needing certain control of the present.

**POPPER:** Always. Without exception in the historical record. The man who knows where history is going requires, by the inexorable logic of his own certainty, the authority to steer everyone there, and to suppress as dangerous the doubters who might slow the journey. I do not say the doomers are tyrants. I say their epistemology is the tyrant's epistemology, and that good people holding a tyrant's epistemology have done much of the worst damage in history precisely because they were good and therefore felt licensed. The cure is the same cure as always: demand the test. Ask the prophet what would prove him wrong, and watch what happens to his face.

**EDO SEGAL:** I want to slow us down, because something rare just happened and the reader should feel its weight. You did not agree about the doom. You agreed about the *shape of the danger* and split cleanly on its content. Karl, you carved out a real exception to your own philosophy — the unsurvivable error where you can't learn from refutation — and Pedro accepted he'd been too glib about it. And then you both turned and identified the same disease in the doom *as practiced*: an unfalsifiable prophecy doing the work of power. So let me ask the hard, naive version, the one the parent at the kitchen table actually asks: should I be afraid? Not of the metaphysics. Tonight, for my kids. What's the true thing to be afraid of?

**DOMINGOS:** Be afraid of stupid systems trusted past their competence making decisions about your life — your loan, your job, your kid's school placement — with no one accountable and no appeal. That's here, it's real, and it's boring, which is why it doesn't trend. Don't be afraid of the machine waking up. Be afraid of us falling asleep at the controls of one that never will.

**POPPER:** And be afraid of the day you stop asking *what would show this wrong* — of any of it, the hype, the doom, the loan decision, the confident answer on your screen. The machine will not enslave you. Your own surrendered judgment might. The open society dies the day its citizens prefer a confident answer to a testable one, and that day is not announced. It arrives quietly, one unchecked fluency at a time.

**EDO SEGAL:** Hold both of those — they go home with the reader tonight whether the metaphysics ever resolves or not. Next round we go back under the hood, to the architecture Pedro spent his life on — the five tribes, the master algorithm — and the question of whether a machine that could learn everything would understand anything. After this.