The Bubble, the Hype, and the Manufacture of Belief

EDO SEGAL: Pamela, you have a credential almost no one in this debate has: you helped build a bubble and you watched it burst. The Fifth Generation in 1983, the expert-systems boom, the winter that followed. So you're not a detached skeptic mocking believers — you were a believer, you felt the pull, and you lived to name the cycle. Timnit, you're trying to puncture the current bubble in real time, and being cast as a naysayer for it. I want you two to compare notes across forty years. Pamela first: what does it feel like, from the inside, to be inflating a bubble you can't see?

McCORDUCK: It feels like clarity. That's the terrifying part. When you're inside the surge, the extrapolation feels not just reasonable but obvious — Simon's prediction that machines would soon do everything a man could do didn't feel like hype to him; it felt like arithmetic. When Feigenbaum and I wrote that Japan might seize the commanding heights of the knowledge economy, we weren't lying or cynical. We were swept up, and being swept up feels exactly like seeing clearly. That's the lesson I'd give Timnit and it's not a comfortable one: the people inflating the current bubble are not, mostly, frauds who know better. They're true believers, and their sincerity is precisely what makes the bubble inflate, because sincere conviction is more persuasive than any lie. The wall arrived anyway. It always does. The dream is real and the progress is real, but the timeline is almost always wrong and the specific shape almost always surprises even the experts. I learned that the hard way, with my name on the book.

EDO SEGAL: Timnit, the historian just told you the inflators are sincere, not cynical. Does that change your critique, or sharpen it?

GEBRU: It sharpens it, and I want to honor how useful that is, because it corrects something people get wrong about my position. They think I'm accusing the industry of lying. Mostly I'm not. The hype works precisely because it's largely sincere — because the gold-rush belief that gold is there is genuinely held by most of the people spreading it. But here's where I diverge from Pamela's framing, and it's the crux. She says the bubble bursts on its own — the wall arrives, the winter comes, the cycle self-corrects. That was true when the technology didn't work very well. It is dangerously less true now, because the systems are useful enough to entrench before the wall arrives. The expert systems of the eighties failed cleanly and visibly — they hit brittleness and everyone could see it. Today's models don't fail cleanly. They produce fluent, confident falsehood — what the industry euphemistically calls hallucination — and the fluency is the product, so the failure looks like success to the people selling it. The bubble doesn't burst. It calcifies, with a handful of firms owning the infrastructure, and the harm gets locked in before the correction comes. The winter saved you, Pamela. I'm not sure the winter is coming this time, and that's worse, not better.

McCORDUCK: That's a genuinely frightening amendment and I think it might be right, which costs me something to say. The winters did function as corrections — brutal, wasteful, but real. Funding fled, hubris deflated, the field was forced to become honest again. If you're telling me this surge is useful enough to skip the winter, to entrench without ever being humbled by a wall, then the one mechanism I trusted to keep the field honest — the burst that punishes overclaiming — is gone. I lived through the winters as disasters. You're making me realize they were also immune systems. And a technology that's learned to suppress its own immune system is a different and more dangerous animal than the one I chronicled.

GEBRU: Yes. And notice what suppresses the immune system: the story. The eschatology of superintelligence, the language of inevitability, the statements about existential risk — all of it functions to make questioning the technology look like standing in front of history. If the machines are about to become godlike, then the documented present harms look trivial beside the imagined future, and skepticism looks like a failure to see the future. The hype isn't froth on top of the real engineering. It's a political instrument that directs where money and authority flow, and it systematically advantages the largest players who can most credibly promise the most spectacular futures. The winter doesn't come because the story has been engineered to prevent it — to convert every doubt into heresy and every critic into a Luddite standing in the river telling the future to stop.

McCORDUCK: And here's where the historian can hand you a weapon, Timnit, because I have one. The single most effective antidote to that engineered inevitability is memory. Someone who knows the field has dreamed this exact dream before — who knows that "this changes everything" has been said at the crest of every wave for seventy years, who knows Simon said it and the Fifth Generation said it and the neural-network revivals said it — is much harder to stampede. The long view, used honestly, is not a fatalism. It's an inoculation against the breathlessness. You want to break the spell of inevitability? Teach people that the spell is recurrent. The breakthrough of the season has happened sixty times. Most of the people who told you it changed everything were sincere, and most of them were early by decades or wrong about the shape entirely. That knowledge is armor, and it's the one thing my profession can give yours.

GEBRU: [pause] That's the best case anyone's made tonight for the historian in the room, and I'll take the armor. The long view as inoculation against inevitability rather than as alibi for it — if that's what you're offering, we're on the same side of this one. My only addition is that the inoculation has to name names. "The field has dreamed this before" is armor. "These specific firms are dreaming it now, for these specific profits, and here's who pays" is a weapon. Armor keeps you from being swept away. The weapon lets you actually redirect the current. You give me the first. I want both.

EDO SEGAL: Let me press on the place where your two corrections of my river metaphor actually conflict, because I don't think they fully reconcile and I want the friction visible. Pamela, your inoculation says: calm down, this has happened before, the breathlessness is recurrent, don't be stampeded. Timnit, your weapon says: act now, these specific people are doing specific harm, the window is closing. But those two counsels point in opposite directions in a crisis. The historian's "this has happened before" can become a reason to wait. The activist's "this is urgent" can become a reason to panic. So which is it — is the long view a brake on Timnit's urgency, or fuel for it?

McCORDUCK: That's a genuine tension and I won't paper it over, because Edo's right that "calm down, it's recurrent" can curdle into complacency, which is its own failure mode and arguably the one I'm most prone to. Here's how I'd resolve it, though. The long view should make you calm about the timeline and urgent about the stakes. Calm that the superintelligence is not arriving next Tuesday, that the breathless extrapolation is almost certainly wrong about when and how. Urgent that the harm Timnit documents is happening right now, in the present, to real people, regardless of where the technology is headed. The mistake is to let calm about the future become calm about the present. The misclassified woman is not a long-term risk. She was harmed last year. My long view should never be used to defer her, only to keep us from being stampeded into the wrong remedy by a manufactured emergency.

GEBRU: And that's the synthesis I can actually use, because it dissolves the false choice the industry forces on us. They want the only two options to be "the future is glorious, accelerate" or "the future is catastrophic, accelerate carefully" — both of which are about the future, both of which defer the present. Pamela's giving me a third stance: skeptical about the prophesied future and urgent about the documented present. That's exactly where I've always tried to stand, and people misread it constantly — they think because I dismiss the superintelligence eschatology I must think the technology is harmless. No. I think the cosmic story is a distraction and the present harm is real and urgent. The long view, used your way, is what lets you hold both without contradiction. It's the most useful thing the historian gives the activist: permission to ignore the prophecy without ignoring the wound.

EDO SEGAL: That's convergence four, and it might be the alliance of the night: memory as inoculation against the engineered inevitability, with Pamela supplying the armor and Timnit supplying the names. Hold that. We're at the top of the second hour, and I want to go to the place where Pamela's whole life is most exposed — her confessed regret, the warning she gave too late, and whether wonder can ever see the harm in time. The candle, and what it costs to love a technology. After this.