Opening Positions

McCORDUCK: Thank you. I want to start a very long way back, because the distance is the argument.

In every culture I could find, going back as far as the records go, human beings have told stories about making an artificial mind. The bronze giant Talos, forged by Hephaestus to guard Crete. Hephaestus's golden handmaidens, who could think and speak. Pygmalion's ivory statue, warmed into life by longing. The brazen heads of medieval sages, said to answer any question. The Golem of Prague, animated by a holy name on parchment, who protected the ghetto and then exceeded his maker's control. These are not idle decorations on human culture. They are, in my reading, rehearsals. We kept imagining the thinking artifact long before we could build one, the way a body sometimes prepares for a motion it has not yet made. And when I sat with Newell and Simon at Carnegie Mellon, with McCarthy and Minsky, with Feigenbaum — brilliant, difficult men who genuinely believed they were a decade from replicating the human mind — I understood that they were the heirs of those storytellers, whether they knew it or not. Mostly they did not. They thought they were making a clean break with everything soft and mythological. They were standing on a much older foundation than they realized.

Why does this matter? Because the persistence of the myth tells you something the engineering can't. We do not tell three thousand years of stories about the hammer or the wheel. Tools don't haunt us. The fact that this particular project — the made mind — has haunted us across every civilization tells me it touches the deepest question we have: what are we, whether we are special, whether mind is the kind of thing that can be made outside a body. The awe people feel before a chatbot tonight is not marketing. It's the return of the oldest feeling we have about our own ingenuity, the vertigo of the mirror that talks back. I spent my life arguing that you cannot understand this technology with engineering alone, that you need the myth, the long memory, the story — and the age of language models, machines whose entire competence is narrative, has vindicated that argument more completely than I ever wanted.

So that is my temple. Not a building any one company owns — a continuity of human longing, the externalizing of our highest capacity, the attempt to make an image of ourselves that can stand apart from us and still be ours. I held this with optimism for most of my life; I called myself, in a book title, a technological optimist, and I meant it as a confession, because optimism in the face of power is a position you have to own up to. But I want to say the rest of it plainly, because I learned it at the end and I owe it to this table. I regret that I did not warn sooner about how the dream would be abused. I was so captivated by the wish to forge the gods that I did not look hard enough at what the forged gods could be made to do in the wrong hands. So my temple has a crack running through it, and Timnit, I expect, is going to put her whole hand into that crack. I'm going to let her.

EDO SEGAL: Timnit.

GEBRU: That was generous, and I want to honor it by being precise rather than polite.

I'll start where Pamela ended, with the abuse, because for me it isn't a crack in the temple — it's the foundation. Let me tell you what I actually see when I look at the systems we're calling AI tonight. I see Gender Shades: I tested the commercial gender-classification products of IBM, Microsoft, and Face++, and instead of reporting one accuracy number the way the vendors did, Joy Buolamwini and I disaggregated by the intersection of skin tone and gender. Under one percent error for lighter-skinned men. Nearly thirty-five percent for darker-skinned women. The single number was a way of hiding. The disaggregation was a way of seeing. And what it let us see is that these systems work beautifully for the people who build them and fail, quietly, billably, for the people who don't.

That's the pattern, and it scales all the way up. A model is a mirror of its training distribution, and a skewed mirror returns a skewed world, with the skew laundered into something that sounds neutral and authoritative. The Stochastic Parrots paper made this concrete: a language model stitches together linguistic forms it has observed, according to the probabilities of how they co-occur, with no reference to meaning. The fluency is real. The understanding is imputed by you, because you are a meaning-making creature and you cannot help projecting a mind behind coherent text. And while you project, the costs accumulate where you're not looking: the environmental footprint of training runs the size of small nations' grids, falling hardest on the communities least likely to benefit; the labor of the people in Kenya and the Philippines who labeled the data and moderated the horror; the work scraped without consent; the capacity of a handful of trillion-dollar firms to deploy systems that touch billions of lives while those billions have no say in whether the thing should exist.

So here is my tower, against Pamela's temple. It is not three thousand years old. It is a concentration of power about fifteen years deep, built by a few corporations, financed by a profit motive, and increasingly justified by a quasi-religious story about a coming superintelligence that will either save us or kill us — a story that conveniently licenses any present harm in the name of an astronomical future. The myth Pamela loves is being used, right now, as the marketing department of that tower. I don't think she's doing the using. I think she handed them a beautiful weapon and they picked it up. My whole career is the argument that the trajectory is not weather, not destiny, not the unfolding of an ancient wish. It is a set of choices, made recently, by people who could choose otherwise — which means it can be contested, and which means we have the agency to decide. That's my opening.

EDO SEGAL: Before I frame the rounds, one discipline I impose on every long conversation because it pays off three hours later. Each of you, in a few sentences — what do you envy in the other's position? Not respect. Envy. The thing her side gets to have that yours doesn't. Pamela first.

McCORDUCK: I envy the floor under her feet. Timnit's position rests on something solid — a body that's harmed or not harmed, a wage that's paid or not paid, an error rate you can measure. She can stand on that and push. My position floats. I'm describing a longing, a continuity of imagination across millennia, and there are mornings when I worry the whole edifice is just a very learned way of being enchanted. She gets to be right or wrong about something. I get to be profound, which is a more dangerous place to live, because you can be profound and useless at the same time and never find out.

GEBRU: And I envy the wonder. I mean it without sarcasm. Pamela gets to stand in front of this thing and feel awe, feel that the universe permits minds to be made, feel part of a story three thousand years long. My discipline requires me to stand in the same spot and ask who paid for the demo and whose face it misclassifies and which worker it traumatized. The vigilance is necessary — somebody has to do it and almost nobody wants to. But nobody should pretend the vigilance is fun. There are days I would trade a great deal to be allowed to just look at the thing and marvel. I'm not allowed. The people I work for can't afford my marveling.

McCORDUCK: That may be the truest thing either of us says tonight.

EDO SEGAL: Two openings and two envies, and you can already see the shape of the evening. It isn't that one of them loves the technology and one fears it. They both fear it. They locate the awe and the danger in opposite places. Pamela says: this is an ancient human story, and the danger is forgetting how old and how mythic it is. Timnit says: this is a recent power grab, and the danger is dressing it up in a story that makes the harm look like fate. Hold both. We start the rounds at the exact seam — the sentence that started this whole series, the one Pamela made famous: artificial intelligence began with the ancient wish to forge the gods. Whose gods? Forged by whom? After the break.