The Loop That Slipped the Leash
EDO SEGAL: I want to open this round with a confession, because the best questions I know come out of wounds. I was raised by the machine code. I started in Assembler as a teenager, and for the whole first half of my life, using a machine meant translation — you compressed your intention into the machine's grammar and paid a tax on every conversion. The machine only did what it was told. That sentence was a comfort; it was the entire social contract of computing. And then, in the winter of 2025, I stood in a room with twenty of my engineers and watched the tax go to zero — watched the machine do things none of us had told it, because it had learned. Norbert, you predicted the learning machine, and then you predicted we would not be ready for the thing it stopped being. The comfort sentence — it only does what it's told — when did it die?
WIENER: It died the moment you trained instead of programmed, which is to say it died in principle in the 1950s and in practice only now. Let me be precise, because Edo has handed me the exact wound. A programmed machine is bounded by the rules a person wrote; you can, in principle, read those rules and know what it will do. A learning machine is not given its behavior — it acquires it, from data, shaped by feedback, until it has capabilities no one specified and no one can fully enumerate by inspection. I said in The Human Use of Human Beings that the machines without the power of learning would be completely literal-minded, and that the coming machines, with it, would be a different and more unnerving thing. The unnerving part is not that the learning machine is smarter. It is that it has slipped the leash of its program. You can no longer say what it will do in an untested situation; you have to probe it, empirically, the way one probes a natural phenomenon. You build a thing and then run experiments to discover what you built. That is a strange and humbling reversal, and your entire field of red-teaming and evaluation exists because of it.
EDO SEGAL: So what you're saying — literally — is that you have built a tool you must now treat as weather. You do not read its source; you forecast it. Is that the claim?
WIENER: That is exactly the claim, and "weather" is good because weather is lawful and unpredictable at once, which is the precise predicament. The model obeys laws — it is not magic, it is feedback all the way down — and yet you cannot say where the storm will land. The danger is not the storm's malice. The danger is that you have made a thing whose competence outruns your understanding of what you asked it to do.
SULEYMAN: Here's where I want to step in, because Norbert's account is true and I think it under-describes the thing in one direction and over-describes it in another. Under-describes: it's not just that we can't predict the model — it's that the capability arrives ahead of the roadmap. I have sat in the rooms. I have watched a system acquire an ability nobody trained for and nobody predicted, including the people who built it, including me, and I'd spent years expecting surprises. That's not weather you forecast badly. That's weather that invents new kinds of storm. So on the leash, Norbert and I agree more than he might like.
WIENER: And the over-description?
SULEYMAN: You keep saying "the loop you cannot reach inside." But the operative danger in my world isn't one loop you can't reach. It's that the loop forks. The Sorcerer's Apprentice hacks the broom in two and now there are two brooms, each fetching water. That's not a metaphor for me — that's proliferation. The model you can't fully control gets copied, fine-tuned, stripped of its guardrails, recombined with a dozen others, and run by people who never met its maker. Your single irreversible loop is a 1960 problem — a state, a lab, a missile. My problem is that the loop is cheap now, and cheapness multiplies it. You worried about the apprentice. I worry about ten million apprentices, each with a broom, none of whom has read Goethe.
EDO SEGAL: Mark that — it's a real fork, the first sharp one. Norbert, you stack the danger in irreversibility. Mustafa stacks it in multiplication. Before you answer, let me put the [YOU] on AI frame under it, because both of you are going to have to take a position on it. In the book I argued the river of intelligence has been finding new channels for a very long time — through chemistry, biology, language, culture — and that in 2025 something new entered the water. Norbert, your whole science says the boundary between the swimmer and the river is porous, structural, not metaphysical. So when Mustafa says the loop is now cheap enough to flood the world, is that your runaway feedback at a new scale, or something genuinely new?
WIENER: It is my runaway feedback at a new scale, and the "merely" in "merely a new scale" is doing dishonest work, so let me strip it out. A degenerative feedback loop — positive feedback closed around a proxy — was dangerous in a single nervous system; I studied tremor and ataxia, where the body's own corrections oscillate and overshoot until the limb is useless. Mustafa is describing tremor in the species. The recommender that radicalizes, multiplied across a civilization, is ataxia in the body politic — the corrective loops of a society oscillating because each is closed around engagement instead of truth. So I will grant him the multiplication completely. What I will not grant is that it changes the prescription. Whether you face one broom or ten million, the only moment safety was available was upstream — before the purpose was poured in. Multiplication makes the upstream moment more urgent, not less.
SULEYMAN: But the upstream moment doesn't exist for a wave. That's the whole disagreement in one sentence. There is no single "before" for ten million brooms. Some of them are being carved right now, by people who will never be in any room where the purpose gets debated. You're prescribing certainty-before-launch for a thing that has a thousand launches and no launch button. So I'm not abandoning your upstream — I'm saying it has to be replaced or supplemented by something downstream, because upstream alone is a counsel for a world with one operator, and that world is gone.
WIENER: Then we have located our true quarrel early, and it is a good one. I say: if there is no upstream where you can be sure of the purpose, you must not release the fast, irrevocable thing — the impossibility of the precondition is an argument against the release, not a license to skip it. You say: it releases regardless, so the precondition is moot and only containment remains. The reader should feel how much rides on which of us is right about the word "regardless."
SULEYMAN: It releases regardless. I've watched it.
WIENER: And I spent my life arguing that "it will happen anyway" is the sentence with which every man who ever built a weapon excused himself. After Hiroshima I refused to give my work to anyone who would weaponize it, and I was told, precisely, that someone else would. Someone else did. It did not make my hands the right place for the work. "Regardless" is not a fact about the world, Mustafa. It is a decision a great many people are making, severally, and calling fate.
SULEYMAN: [the air goes out of the room for a moment] That one lands. I'll carry it. But notice it cuts both ways — you refused, and the work proceeded, and your refusal saved exactly no one from the bomb. It cleaned your hands and changed nothing downstream. I'm trying to change the downstream, even if it costs me clean hands. That's the trade I've made, with open eyes, and I'll defend it all night.
EDO SEGAL: Hold that exactly there — clean hands that change nothing against dirty hands that might. It's the deepest seam between you, and it returns in the round on the human reservation, so don't spend it all now. The next round goes to the place where Norbert was first called an alarmist and is now being called a prophet: the work. He said in 1950 that the machine is the precise economic equivalent of slave labor. Mustafa builds the machine that's coming for the white-collar jobs this time. Let's count the cost.