Against the Hype, Against the Oracle

**EDO SEGAL:** Pedro, you are famous for a sentence I want to put in the room: the real danger is not that computers will get too smart and take over, but that they are too stupid and have already taken over. And you've spent the whole boom as one of deep learning's most stubborn skeptics — not denying the achievements, refusing to let them be inflated. Karl, you spent your life on why the inflated, unfalsifiable theory is so much more attractive than the modest, testable one. So I want to watch the two great deflaters of this era find out whether they're deflating the same balloon. Pedro, what is the hype, mechanically, and why does it win?

**DOMINGOS:** The hype is the confusion of fluency with understanding, and it wins because fluency is the exact thing humans evolved to read as understanding. A language model predicts the next token with uncanny skill, having absorbed the statistical structure of everything we've written. The output reads as intelligent — but the reading is *ours*. The system has no model of meaning, no grasp of truth, no idea what its words are about. It's a savant: dazzling narrow ability welded to a stunning absence of common sense, capable of feats no human could match and of errors no human would make. And the absence of common sense is the crux, because common sense is the vast ocean of unstated knowledge about how the world works that we acquire effortlessly and machines lack entirely. That's why they fail outside their distribution without warning — the savant has no floor under it.

Why does the hype win? Because it serves everybody at once. It serves the companies raising money. It serves the journalists who need a story bigger than "improved autocomplete." And — Karl will like this — it serves the doomers, because a public convinced the machine is nearly superhuman is a public primed to fear it. Hype and doom are the same balloon inflated from two ends. Both need the machine to be more than it is. I oppose both, and I've made enemies on both sides for it, which is how I know I'm roughly in the right place.

**POPPER:** I find very little to disagree with and I want to add the mechanism underneath your mechanism, because you have described *what* wins and I spent my life on *why* it must. The inflated claim wins because it explains more. That is its fatal attraction. The modest, testable claim explains a little and forbids a great deal — it says "this and only this, and here is what would refute me." The inflated claim says "everything," and forbids nothing, and so it can never be embarrassed by events. Marxism explained every strike and every absence of a strike. The hype around your machine explains every success as intelligence and every failure as a temporary limitation soon to be scaled away — notice that move, Pedro, it is the purest unfalsifiability: *the next model will fix it*. That sentence cannot be refuted, because there is always a next model. It is the perpetual-motion machine of belief. And a population fed on it loses the very faculty I cared about most, which is the disposition to ask of an impressive thing: yes, but what would show you wrong? When confidence is cheap and instant, that question feels rude, then unnecessary, then unthinkable. That is how an open society loses its mind — not to a tyrant, but to a thousand fluent answers that made doubt feel like bad manners.

**DOMINGOS:** "The next model will fix it" — you've nailed the unfalsifiable core of my own industry's faith and I'm going to be quoting that back at conferences, with attribution. Although I'll defend one version of it, because here's where I split from you. Sometimes the next model *did* fix it, and that's not nothing, that's the most reliable empirical regularity my field ever found — the [scaling pattern](https://www.youonai.ai/fieldguide/med/scaling_laws), capability rising with data and compute across many orders of magnitude. So when I say scale might fix a limitation, I'm not always doing the Marxist dodge. Sometimes I'm reporting a curve that's held for a decade. The honest position, which is murderously hard to hold in public, is: scaling is a real and astonishing trend, *and* a trend is not a law, *and* the people selling it as a law are doing exactly the thing you're describing.

**POPPER:** And there — you have said it better than I could have hoped — *a trend is not a law*. That single sentence is the antidote to the whole disease, and it is the sentence I spent two books defending against the [historicists](https://www.youonai.ai/fieldguide/med/historicism) who said history had a direction you could read off and ride. The scaling curve is a trend. It has held. It may continue. It may break tomorrow, for reasons no extrapolation from the curve can anticipate, because the future growth of knowledge is precisely the thing that cannot be predicted from present knowledge — that was my one argument against all prophecy, and it survives intact. The man who tells you the curve is destiny has converted a trend into a law and a decision into a fate, and the moment you accept the fate you have handed away your agency to whoever is drawing the curve.

**DOMINGOS:** Which is, not coincidentally, whoever owns the data centers. We'll get to that. But yes — I refuse the law and I keep the trend, and I get attacked by both the cult that thinks the curve is God and the cult that thinks it's a fraud. Your demarcation line, Karl, drawn through my own field, would clear out about eighty percent of the public conversation about AI, and the field would be healthier for it.

**EDO SEGAL:** I have to put my own ledger on the table here, because I am implicated. I built engagement machinery, years ago — systems tuned to capture attention, and I knew the loops, and I told myself a story about value while the metrics climbed. So when Pedro says the hype serves everyone who profits from it, I am not above that table. I have sat at it. And what I learned, expensively, is that the inflated story doesn't feel like a lie from the inside. It feels like vision. The man drawing the curve as destiny usually believes it. That is what makes it so hard to refute and so dangerous — Karl's oracle is sincere. So let me ask the thing under the thing, to both of you, briefly: if hype and doom are one balloon, what is the *modest true claim* it's crowding out — the testable, forbidding, unglamorous thing we should be saying instead?

**DOMINGOS:** That these are extraordinary, narrow, brittle tools that amplify whoever wields them, that the gains and the harms are both real and both mundane, and that the question isn't when the machine wakes up — it never does — but who's accountable when the stupid system, trusted past its competence, ruins a life at scale. That's the claim. It forbids a lot. It doesn't sell.

**POPPER:** And I would add only: that every output is a conjecture awaiting its test, including this one, including mine. The modest true claim is that we do not know as much as the fluency suggests, and that saying so out loud, repeatedly, against the current, is the entire civic duty of a thinking person in your age. It is not a thrilling banner. But the open society was never built on thrilling banners. It was built on people willing to say *I might be wrong* and mean it.

**EDO SEGAL:** Mark the third convergence, and it's the one I least expected to be unanimous: hype and doom are the same inflation, and the casualty of both is the modest, testable, forbidding claim — the one that says we know less than the machine's confidence implies. Two of the great skeptics of this era just shook hands across a century. Hold it. Next round, we follow the money the hype is raising, because a trillion dollars has moved on the strength of these claims, and Karl has a name for what happens when prophecy becomes policy. The death cross, after this.