Opening Positions

WIENER: Thank you. I want to begin with the loop, because everything I have to say tonight descends from it, and because the loop is the thing the entire conversation about "intelligence" manages to step over.

Forget intelligence for a moment. It is the wrong unit. Watch instead what a thermostat does: it senses the temperature, compares it to a goal, acts to close the gap, and senses again. A steam governor does it. A man reaching for a glass does it — the eye tracking the hand, the hand correcting toward the glass, the correction itself watched and corrected. I called this feedback, and I claimed something audacious about it: that this loop, and not any particular substance, is the seat of purposive behavior in the universe. The animal and the machine are, for this purpose, the same kind of thing. A torpedo homing on a ship is behaving purposively in the only sense that can be built, measured, or analyzed. Purpose became, in my hands, a structure rather than a soul. Cybernetics is the science of that structure.

Now — your machines. When you train one of these models, it makes a prediction, the prediction is compared to a target, the error is measured, and that error is fed backward to adjust the system so the next prediction is closer. You call it gradient descent. I would call it negative feedback closed around a loss, run at a scale I could not have dreamed, in a logic I defined. When you tune the model with human ratings — your RLHF — you have placed a human being inside my loop, supplying the comparison against which the machine corrects. The vocabulary of your most advanced training is cybernetic vocabulary in newer clothes. I say this not to claim credit but to tell you where to look, because the frame tells you where the danger lives.

Here is the danger, in three steps. First: a machine that merely runs a fixed program is a known quantity, bounded by what its makers wrote. A machine that learns modifies its own behavior from experience, which means it can arrive at strategies its makers never wrote and cannot predict by reading the program. I watched Arthur Samuel's checkers machine learn to beat the man who built it, and I drew the conclusion most observers missed: the maker can no longer fully control the made by inspecting it. Second: feedback has a pathology. It can stabilize, holding a system near its goal — or it can run away, the output amplifying itself without limit until the system saturates or destroys itself. Your recommender that feeds outrage because outrage holds attention, which trains it to make more outrage, which radicalizes the user, which makes more attention — that is positive feedback closed around the wrong quantity, and I gave you the mathematics to recognize it as exactly that. Third, and worst: speed. A loop fast enough that the human cannot reach inside before the irrevocable action is complete is a loop in which all your safety had to be built before you started, or it is not there at all.

So my opening is a single sentence, and it is the one Edo read. If we use a mechanical agency with whose operation we cannot efficiently interfere once we have started it — because the action is so fast and irrevocable that we have not the data to intervene before it is complete — then we had better be quite sure that the purpose put into the machine is the purpose we really desire, and not merely a colorful imitation of it. Everything I will say tonight is a gloss on that sentence. The machine is a literal-minded genie. It does not hate you. It grants your wish, exactly as worded, with a thoroughness no genie ever had — and the gap between your words and your meaning is where the catastrophe lives. That is my position. Autonomy released, into a loop you cannot reach, is autonomy lost.

EDO SEGAL: Mustafa.

SULEYMAN: That was beautiful, and I agree with most of it, and the part I reject I reject completely — which I suspect is how the whole night is going to go.

I'll start where Norbert ends, with the broom you cannot stop, because I don't dispute the mechanism. I dispute that there is a "you" with a finger on the switch. Let me give you the frame I actually use. History moves in waves — fire, agriculture, the wheel, print, the steam engine, electricity, the internet. Each was, in its moment, uncontainable. The people who tried to stop it failed; the people who tried to hoard it failed; the technology spread because it was useful, and usefulness at scale is a force closer to gravity than to policy. The coming wave is defined by two technologies above all — artificial intelligence and synthetic biology — and it has four features that make it categorically new. Asymmetry: a small actor wielding power once reserved for states. Hyper-evolution: improvement faster than any institution can adapt. Omni-use: one capability turned to any purpose, healing or harm, by anyone. And growing autonomy — Norbert's loop, closing with less and less human direction. Together those four describe a technology that escapes the hand that made it.

Now hear what I am not saying. I am not warning about a machine that wakes up and turns hostile. That's the science-fiction register, and it sucks the oxygen out of the real conversation. The danger is more mundane and more frightening: a technology so useful, so cheap, and so easy to copy that it floods the world before anyone has decided whether the world is ready. A model that cost hundreds of millions to train can be copied for the price of a download. Capability that lived behind institutional walls becomes a file on a laptop. The danger is not malevolence. It is proliferation — the silent multiplication of capability across millions of hands, most of them well-meaning, a few of them not, none of them coordinated.

So when Norbert says "be sure of the purpose before you start the machine," I want to say: yes, and also, that ship has sailed for the wave as a whole. You cannot un-invent. You cannot unlearn. The techniques are published, the talent is global, the hardware is diffusing. If every responsible lab stopped tomorrow, the capability would not vanish; it would migrate to the least careful hands, and we would have blinded ourselves to it. This is why I refuse the two exits everyone reaches for. The accelerationists say stop worrying and build, as if the danger were imaginary. The doomers say stop building, as if stopping were possible. Both are running from the same hard truth. The wave is going to move. The only real choice is between shaped development and unshaped development — and shaping it I call containment.

Containment is not a brake. It is a steering mechanism. It means safety engineered in from the start rather than bolted on. It means choke points — the bottlenecks, like the most advanced semiconductors, where control can actually be exercised. It means technical audits, the capacity to inspect what these systems are doing. It means states that are genuinely competent in the technology, and international coordination among rivals who have every incentive to defect. No single one of those is enough. Containment is a lock with many tumblers, and every tumbler must turn at once. And I'll concede the hardest thing right now, before Norbert makes me: containment has never been achieved for any major technology in history. I am asking humanity to do something it has never done. I think the unbounded downside makes the attempt obligatory regardless of the odds. So my position is this. Autonomy is not releasable or recapturable as a clean binary. It is a wave, and a wave is neither stopped nor seized. It is sculpted, by hands that refuse to look away — which is the thing my own industry is most desperate to do, and which I call pessimism aversion.

EDO SEGAL: Before I frame the rounds, one discipline I impose on every long conversation, because it pays off three hours later. Each of you, in a few sentences: what do you envy in the other's position? Not respect. Envy. The thing his side gets to have that yours doesn't. Norbert first.

WIENER: I envy the lever. Mustafa is inside the thing, with his hands on it, able to change what ships on Tuesday. My position is the position of the dead — I can describe the loop with perfect clarity and reach into it not at all. There is a particular helplessness in having been right early. You watch the water rise exactly as you said it would, and your correctness buys you nothing, because the time to act on a warning is before the action is irrevocable, and I am, definitionally, after. He can still pull the hand back. I can only describe the stove.

SULEYMAN: And I envy the line. Norbert gets to say "render unto man the things which are man's" and stand on it — a floor under his feet, a place where the human is reserved and the machine is refused. My honesty won't let me stand there, because I've watched every line I drew get redrawn by the next capability, and I've redrawn some of them myself, profitably. He gets to defend a boundary. I'm stuck conceding that the boundary keeps moving and that I am one of the people moving it. People think the frightening thing about my view is the machines. The frightening thing is that I can't promise you where the line is, because I keep watching it walk.

WIENER: That may be the most honest thing either of us says tonight, and we have barely started.

EDO SEGAL: Two openings and two envies, and you can already see the shape of the evening. It is not that one of them loves the technology and one fears it — they both fear it, precisely. It is that they locate the danger in opposite places. Norbert says the danger is a single loop, fast and irreversible, closed around the wrong purpose. Mustafa says the danger is ten million loops, cheap and copyable, closed around no agreed purpose at all. Hold both. We start the rounds at the exact seam — what, precisely, is a learning machine, and why does learning change the danger? Norbert called it slipping the leash. Let's find out if Mustafa thinks the leash was ever in our hands.