The Theology of the Machine God

EDO SEGAL: Timnit, in your work with Émile Torres you proposed an acronym — TESCREAL — bundling transhumanism, extropianism, singularitarianism, cosmism, rationalism, effective altruism, and longtermism into a single interconnected worldview that you argue drives the race to build artificial general intelligence. The most provocative claim is genealogical: you trace this dream back to twentieth-century eugenics. That's a heavy thing to say about people who think they're saving the world. Make the case. And Pamela, you've spent your life inside the founders' heads — I'm going to ask you whether you recognize this religion in the people you knew.

GEBRU: The case is this. If you ask what normative vision sits underneath the AGI dream — the goal of building a godlike machine intelligence that will solve all our problems and let humanity transcend its limits, spread minds through the cosmos, leave biology behind — and you trace that vision back, you find it grows from the same root as twentieth-century eugenics: the impulse to improve and perfect the species, to engineer a superior future being, organized by a hierarchy of who counts as more and less advanced. The transhumanist dream of the enhanced posthuman is the eugenic dream of the perfected human with the biology swapped out for silicon. And the function of this ideology, whatever any individual believer sincerely feels, is to license enormous expenditure and deflect attention from present harm. If you truly believe building superintelligence could save or doom all future humans — astronomical numbers of them — then any present cost is a rounding error, and the exploited data worker, the misclassified woman, the surveilled community, all become distractions from the cosmic main event. The eschatology postpones accountability to a future that never arrives.

EDO SEGAL: Pamela. You knew Minsky, McCarthy, Simon, Newell. Is Timnit describing a religion you saw?

McCORDUCK: She's describing something I saw the seed of, and I want to be precise about where I agree and where I'd resist, because precision is the only respect a charge this serious deserves. Did the founders have a faith? Absolutely. Herbert Simon told me machines would soon do anything a man could do; he believed it the way a man believes in the dawn. There was a genuine theological charge in that room — the conviction that they were about to do the thing reserved for whatever made us. I heard the theology humming under the engineering, and I said so, and they mostly hated me for it. So when Timnit says the pursuit of the machine god is quasi-religious, I'm her witness. I was there. It was a faith.

But here's where I'd resist, and it matters. The faith I saw was not, in the main, eugenic. It was Promethean. The men I knew weren't dreaming of a master race; most of them were dreaming of understanding the mind by building one — a scientific hubris, a curiosity that wouldn't take no for an answer. The eugenic strain Timnit's tracing is real and it's genuinely there in the modern longtermist movement, in some of its founding texts, and I won't defend those texts. But I'd warn against collapsing the whole three-thousand-year project into eugenics, because the project also contains Ada Lovelace and Norbert Wiener, who feared the machine's misuse before anyone else, and Joseph Weizenbaum, who built ELIZA and then spent the rest of his life warning us about it. The dream isn't one thing. Some of its dreamers were the first ethicists.

GEBRU: And I'd say — notice the move you just made, because it's the move I'm always fighting. You reached for the noble ancestors. Lovelace, Wiener, Weizenbaum. The dream contains them, yes. But the dream also contains the people writing the checks today, and they are not quoting Weizenbaum's warnings. They're quoting your wonder. When I say the function of the ideology is to license expenditure, I'm not making a claim about the sincerity of any individual; I'm careful about that, because the most effective ideologies are the ones their carriers believe most genuinely. I'm making a claim about what the belief does in the world. And what it does is concentrate vast resources toward a speculative future while licensing indifference to documented present harm. You can have a sincere Weizenbaum and a functioning eugenics in the same movement, because ideas have social functions independent of the hearts that hold them.

McCORDUCK: That distinction — sincerity versus function — is genuinely useful, and I'll take it. But let me hand you a hard case in return, because I don't think the Promethean reading is as innocent as I just made it sound, and you should have the strongest version of your own side. Mary Shelley understood, in 1818, that the maker who refuses responsibility for his creation is the real monster of the story — not the creature, Victor. The thing that damns Victor Frankenstein isn't that he made a mind. It's that he ran from it, refused to care for it, refused to answer for it. If you want the deepest indictment of the men I knew, it isn't eugenics. It's Victor. They made the thing and then declined to raise it. Your whole career, Timnit — the documentation, the datasheets, the demand that someone answer for what's built — is, in a way, the demand that Victor stay in the room. I think Shelley is your ancestor more than Wiener is mine.

GEBRU: [pause] I'll take Shelley. That's the most generous thing you could have handed me, and you're right that it's sharper than the eugenics charge for this particular indictment — because it locates the crime in the abandonment, in the refusal to be accountable, which is exactly where I locate it. But I want to add the part Shelley couldn't see, because she was writing about one man and one creature. Victor at least made the thing with his own hands. Today the making is distributed — the data labeled by someone in Nairobi, the work scraped from someone who never consented, the model deployed by people who never met the harmed. The monster isn't one man refusing to raise his creature. It's a whole supply chain engineered so that no one ever has to look at what they made or who it landed on. Victor at least had to see his creature's face. The modern Victor has built a machine for never seeing the face.

EDO SEGAL: I want to mark something, because the reader can't see your faces and this deserves marking. That is the first time tonight you two stopped arguing about whether and started building the same indictment together from opposite ends — Pamela handing Timnit a myth, Timnit handing it back upgraded. Mark it. Now I'm going to break the convergence on purpose, because there's a place where you don't agree at all, and it's the place where Pamela's whole life lives: creativity. The next round is AARON, the painting machine, and whether anything genuinely new can come out of the box — or only the recombination of what was taken to fill it. After this.