Who Holds the Pen — Power, Google, and the Funnel

EDO SEGAL: Timnit, in late 2020 you were co-lead of Google's Ethical AI team — a sharp internal critic at the heart of a company whose business depended on what you scrutinized. The parrots paper went through internal review, management demanded you retract it or strip the Google-affiliated names, you asked the obvious questions — who reviewed it, what was deficient, by what process — and within a span of weeks you were out, your access cut while you were on vacation. I'm not going to ask you to relitigate a personnel file. I'm going to ask the structural question. What does that episode tell us about whether this field can produce honest knowledge about itself?

GEBRU: It tells us the field's knowledge runs through a funnel with a corporate valve on it, and everyone working in it knows exactly where the valve is. Understand what the paper was: a survey. A literature review with citations, cataloguing risks the company's own products carried. Not fraud. Not error — no one ever identified the error, then or since. For writing a survey, the leading corporate lab in the world pushed out two of its most prominent researchers, both women, both of whom had built its ethical credibility. And here is the mechanism, which is more efficient than censorship: they didn't need to censor anyone else. They demonstrated, once, in public, what happens to a career that itemizes costs. The chill does the rest for free. Then the same companies fund the conferences, license the compute, pay the professors, and hire the graduates, and we are asked to treat the resulting literature as the disinterested record of what these systems are. My critique sounds lonely not because it's rare but because it's expensive, and the price list was published in December 2020 with my name on it.

This is why my deepest concern isn't any single model. It's the concentration of power. The capacity to build these systems now sits with a handful of enormously wealthy firms that command the compute, the data, and the talent, and that answer to their shareholders. They deploy across the whole world, touching billions, and the billions affected have essentially no say in whether the thing is built or how. That gap — between who decides and who is affected — is the defining injustice of this era, and Ray's beautiful curve is the perfect anesthetic for it, because it tells the billions their powerlessness is just the weather.

EDO SEGAL: Ray, you were at Google when this happened. Senior enough that your word carried weight, and the record of senior researchers defending her publicly is thin. I'm not asking you to relitigate it either. I'm asking: does the valve she's describing falsify the science your whole framework rests on?

KURZWEIL: Let me take the part aimed at me first, because dodging it would prove her point. Could people in my position have said more, then, publicly? Yes. I told myself what senior people always tell themselves — that I didn't know the details, that it was a management matter, that my lane was research. I notice those sentences more clearly now than I did then. So I'll hand her that, without an argument attached to it. It happened, it was wrong, and the structural claim it supports — that proximity to revenue bends what gets published — is real. I won't pretend otherwise to win a point.

Where I push back is the inference that the valve falsifies the curve. The capability data isn't hype because corporations fund it. Replication exists. Open-weight models exist, built by people with every incentive to embarrass the labs, and they land on the same curve. Academics adversarially probe these systems daily, and the systems keep passing probes nobody trained them on. Timnit's argument tells you precisely where to point your skepticism — and you should point it, hard. But sociology tells you to audit the record. It doesn't get to be the audit. If the only thing holding up the exponential were a press release, the open-source community would have exposed it years ago. They tried. They reproduced it instead.

GEBRU: I accept the concession about the eyesight, Ray, and it's more than most of your colleagues have offered, so thank you. But you've slid past the actual claim. I never said the curve is fake. I said the meaning of the curve is manufactured. The compute concentrates in a few firms; the few firms decide what gets built and measured; the measurements feed a narrative of inevitability; the narrative justifies more compute concentrating in the few firms. It's a flywheel, and your law is the story the flywheel tells about itself. The open-source models you cite are trained on the same scraped corpora, the same hidden labor, inside the same concentration — they're not outside the system, they're the system's hobbyists. Reproducing a number doesn't reproduce the legitimacy of the choice to build the thing at all.

EDO SEGAL: Stay on the credibility point one more beat, because it's quietly enormous. For years the corporate labs borrowed the legitimacy of the academy — peer review, conferences, the costume of disinterested science. Timnit, you're saying that costume slipped. What does it cost the field if every paper from a corporate lab now has to be read as potentially load-bearing for a product?

GEBRU: It costs the field its claim to be science at all, in the parts that touch revenue. And I want to be careful, because this isn't an accusation that the researchers are liars — most of them are honest people doing careful work. It's structural. When a paper can be suppressed because its conclusions are commercially inconvenient, then every paper from the same source becomes harder to trust, not because the authors are dishonest but because the institution has demonstrated what it will do when the stakes are high enough. You can't selectively distrust only the suppressed papers, because the suppression is invisible — you only see the ones that survived the valve. So the whole corpus inherits the doubt. That's the real cost of what happened to me. Not my job. The credibility of an entire literature that the public, and the regulators, are relying on to understand what these systems are.

KURZWEIL: Then we agree on the flywheel and disagree on whether to unplug it. Because here's what I can't get past, Timnit. The same concentration you fear is, at this exact moment, the only reason a kid in a place with no university can talk to the sum of human knowledge for free. The flywheel that scares you is the democratization you'd be killing. I want to break the concentration of ownership without breaking the distribution of capability, and I think you sometimes aim at the second when you mean to hit the first.

GEBRU: No — I aim at exactly the first, and you keep using the second as its human shield. "Think of the kid" is the oldest move in the book; it's how every extractive industry has ever defended itself. The kid does not need the firms to own the model for the kid to have a tutor. That's a choice they made, not a law you found. Decouple them and I'll celebrate the tutor. Bundle them and call the bundle physics, and I'll fight you for the rest of my life.

EDO SEGAL: I want to name what just happened, because the reader can't see your faces, and that was the first exchange tonight where neither of you was smiling. And notice the strange topology — you both distrust the companies, you both think the public discourse is corrupted, you both want the capability in more hands. You disagree about whether the engine of the danger is the power that owns the machine or the credulity about what the machine is. That fork is the river splitting, and the next round follows the water down to the people standing in it. The merger Ray promises, and the workers Timnit says are already inside the machine. After this.