The Candle, and the Cost of Loving the Thing

EDO SEGAL: I want to open this round with a confession, because the best questions I know come out of wounds, and this one is mine. I built engagement machinery. I knew the loops. I designed systems to hold human attention because attention was the product, and I told myself the usual thing — that it was neutral, that people chose, that I was just lowering friction. I slept fine for years. So when I ask this next question, I'm asking it of myself first. Pamela, in your 2019 memoir you confessed that you spent your career telling the wonder and not enough of the warning — that you were so captivated by the wish to forge the gods that you didn't look hard enough at what the forged gods could do in the wrong hands, and you named deepfakes, synthetic deception, specifically, as the abuse you wished you'd flagged sooner. I want to ask you the cruelest version of the question. Did the wonder prevent you from seeing the harm — or did you see it and choose the wonder?

McCORDUCK: [long pause] The honest answer is the first, and it's worse than the second. If I'd seen the harm and chosen wonder anyway, that would be a moral failure I could at least take responsibility for. What actually happened is more insidious: the wonder organized my perception so that the harm was hard to see at all. When you love a technology — and I did, I loved the field and the people in it with real warmth — you see its promise more vividly than its peril, because the promise is what drew you in. The love isn't a choice you make after seeing clearly. It's a lens that shapes what counts as worth seeing. I wasn't suppressing the warning. I genuinely wasn't perceiving it at full strength, because my attention had been trained, by affection, to land on the marvel. That's the specific failure mode of people who love a powerful thing, and I'm its case study. I published my regret in 2019, right on the edge of the generative flood, and I still didn't see how fast the deception would come.

EDO SEGAL: Timnit, that's a confession about the structure of attention itself — that love is a lens that hides the harm. You've never had that problem. I want to ask you the inverse, and it's not a soft question. Has your vigilance ever been a lens too? Has the discipline of always asking who's harmed ever made you unable to see something real — a genuine marvel, a genuine good — that was right in front of you?

GEBRU: [pause] That's a fair question and I won't dodge it, because it would be dishonest to claim my standpoint has no costs. Yes. The vigilance is a lens, and like any lens it has a periphery it doesn't show me. There are times I've been so trained on the harm that I've been slow to credit a real benefit, slow to sit with something genuinely good because I was already scanning for the catch. I said earlier I envy Pamela's wonder, and I meant it as more than a nice line — the discipline genuinely costs me the capacity to just marvel, and that's a real loss, not a pose. But here's the difference I'd insist on, and it's not symmetric. Pamela's lens, the wonder, was the field's default; it was rewarded, funded, celebrated, and it left the harmed invisible for decades. My lens is the correction nobody wanted, and it makes the harmed visible at the cost of some of the marvel. If we're choosing which lens a powerful field should institutionalize, I'd rather have a generation that's slow to marvel than a generation that's slow to see who's bleeding. The costs aren't equal because the stakes aren't equal.

McCORDUCK: I won't argue the asymmetry — you're right that the field defaulted to my lens and it cost the vulnerable forty years. But let me say something in defense of the candle, because I think you need it more than you admit, Timnit. Edo writes in his book about a candle in the darkness — the small human light that the machine can't replace, the thing worth protecting. I'd say the wonder, the capacity for awe, is part of that candle. A movement that can only see the harm, that has trained itself out of marvel entirely, loses something it needs to win — because people don't fight for a future they can't feel any joy about. Your DAIR isn't only a critique; it's a vision of technology built on a different motive, technology that genuinely helps people. That vision requires being able to imagine a good outcome vividly enough to want it. The candle isn't naive. It's the fuel. Don't let the necessary vigilance blow it out, or you'll win every argument and lose every heart.

GEBRU: [pause] That's the most useful thing you've said to me personally tonight, and I want to receive it as more than a debate point. You're right that refusing inevitability is supposed to be an act of hope, not despair — that the whole reason I insist the future isn't fixed is so it can be better, and you can't fight for better if you can't picture it. I've said it myself: it's humans who decide whether all this should be done, and we have to remember we have the agency to decide. That's not a grim sentence. It's a hopeful one. Maybe I've let the fight for it sound grimmer than the hope underneath it. The vision is real — community-rooted technology, work that helps rather than extracts, the pen shared among the many instead of held by the few. I should let that candle show more than I do.

EDO SEGAL: I want to stay in this confession a moment longer, because I made one too and I haven't paid the rest of my toll. I told you I built engagement machinery. Here's the part I left out. I told myself the wonder story about my own work — that I was connecting people, lowering friction, democratizing reach — and the wonder story was true, which is exactly what made it dangerous, because every true good thing it did was also the thing that hid the harvest. Pamela, you've described the wonder as a lens that hides the harm. I'm telling you it's worse than a lens, because a lens distorts a thing that's there. The wonder doesn't distort the good — the good is real. It just stands in front of the cost so completely that you'd have to actively turn your head to see around it, and the whole system is designed so you never need to turn your head. So my question to you both is the practical one, the kitchen-table one: how does an ordinary person — a parent, a teacher — keep the wonder and still turn their head?

McCORDUCK: You build the turning-of-the-head into a habit, so it doesn't depend on suspicion. The historian's version is: every time you feel the awe, ask who isn't in the picture. Not as an accusation — as a reflex, the way you check a mirror before you change lanes. The awe is allowed. The reflex rides alongside it. I failed for forty years because I treated the two as opposites, as if to feel wonder honestly you had to suspend the suspicion. They're not opposites. They're a pair of eyes. You can keep both open.

GEBRU: And the engineer's version is the same reflex made concrete: ask who built it, on what data, paid how much, accountable to whom, and what happens to the person it fails. You don't have to be a researcher to ask those questions — they're the questions a parent already asks about a daycare, a school, a doctor. We've just been trained to not ask them about software, to treat the machine as if it fell from the sky instead of being built by people with addresses. The candle and the receipt, Pamela said it — and the parent at the kitchen table can hold both, because she already does it about everything else in her child's life. She just has to extend the habit she already has to the glowing box she's been told is magic. Nothing about the box earns an exemption from the questions she'd ask of anything else with that much power over her kid.

EDO SEGAL: I want to mark this for the reader, because it's the warmest moment of the night and it's also the most important. Pamela came in defending wonder and confessed wonder blinded her. Timnit came in defending vigilance and confessed vigilance has a periphery. And right here, they traded — Timnit took the candle, Pamela took the asymmetry. Neither of you put your weapon down. You just admitted what it costs to carry it. That's convergence five, and it's the human one: every lens that lets you see also hides. Hold the candle. Now I have to break the warmth, because there's a place where you don't agree and it's measurable — the benchmark, the number, what the data shows and what it's built to hide. After this.