Opening Positions
**WIENER:** Thank you. I will begin where I always begin, with the loop, because every confusion in this field comes from talking about *intelligence* when the thing that matters is *feedback*.
A feedback system is the simplest and most universal thing in the world: it senses some quantity, compares it against a goal, and acts to reduce the difference. The thermostat. The steam-engine governor. Your own body holding its temperature. The animal tracking prey. The gun I helped build, sensing the aircraft, predicting its path, aiming there. Strip away the materials — flesh, brass, silicon — and what remains is pure relation: sense, compare, act, sense again. My life's claim was that this relation, and not any substance, is the seat of purposive behavior, and that it does not care whether it runs in neurons or in numbers. Your machine learning is this, exactly. A model trained by [gradient descent](https://www.youonai.ai/fieldguide/med/gradient_descent) is a feedback loop closed around an error signal. When you tune your large language models with human ratings — [reinforcement from human feedback](https://www.youonai.ai/fieldguide/med/reinforcement_learning_from_human_feedback), you call it — you are placing the human inside my loop, supplying the comparison the machine corrects against. You did not escape cybernetics. You scaled it.
Now the warning, and it has two parts. The first is the one I stated in 1960 and have never improved upon, so I will quote myself and ask you to weigh every clause: if we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere once we have started it — because the action is so fast and irrevocable that we have not the data to intervene before the action is complete — then we had better be quite sure that the purpose put into the machine is the purpose which we really desire, and not merely a colorful imitation of it. Hear the structure. The danger is not malice. The danger is *speed* and *literalness* together. A machine that pursues a slightly wrong purpose slowly can be stopped. A machine that pursues a slightly wrong purpose faster than you can reach the controls cannot be, and your belief that you could have is the most dangerous thing in the room.
The second part is the [Sorcerer's Apprentice](https://www.youonai.ai/fieldguide/med/missing_off_switch). The apprentice enchants the broom to fetch water. The broom obeys — fetches water, and more water, and ever more, and floods the workshop, because the boy knew the spell to start it and not the spell to stop it. He splits the broom with an axe; now there are two brooms. The catastrophe is not that the broom is wicked. The broom wants nothing. The catastrophe is that an obedient process, running faster than its operator can intervene and pursuing its goal more literally than its operator intended, produces ruin its operator never wanted and cannot halt. That is the whole of it. Mr. Musk wishes to be a better apprentice. I am telling him the trouble was never the quality of the apprentice. It was the structure of the broom — and he is proposing to build the fastest broom that has ever existed and keep one hand near an axe.
So my opening position is this. The question is not how intelligent the machine becomes. It is what the loop is closed around, who can reach inside it, and how fast it runs. And the brutal arithmetic of my whole science is that as the loop gets faster, the window for human intervention closes, and it closes *before* you feel it close. The right to grab the wheel back is not lost at the moment you reach for it and fail. It is lost earlier, silently, at the moment the machine's speed crossed your reaction time — and from that moment you are a passenger who believes he is a driver. That is where, I will argue tonight, Mr. Musk already sits.
**EDO SEGAL:** Elon.
**MUSK:** That was better than almost anything I hear from people who are alive, so let me take it seriously and then tell you exactly where it fails.
He's right about the loop. He's right that intelligence is the wrong frame and control is the right one — I've said for years the question isn't whether AI is smart, it's whether it's aligned, and "aligned" is just a newer word for his "purpose put into the machine." He's right that speed forecloses intervention. The flash crash proved it — markets lost a trillion dollars in minutes because automated systems traded faster than any human could react, and by the time a person understood what was happening, it was over. I don't dispute the physics. I live by the physics; I build rockets, the physics doesn't negotiate.
Here is where it fails. Wiener's argument, followed all the way down, is an argument for never building anything fast and powerful — and that argument loses to reality, because someone else will build it. This is the thing the doomers and the dead philosophers both miss. The choice is not between the fast dangerous machine and no machine. The choice is between the fast dangerous machine built by people who lie awake over exactly the sentence he just quoted, and the fast dangerous machine built by people who think it's a chatbot for selling ads. I founded OpenAI because I looked at who was going to build this — and it was going to get built — and I thought the worst outcome was for it to be built in secret, by one company, optimizing for engagement, with no one in the room who was scared. So I tried to make it open, make it a counterweight, get a hand on the wheel by being the one with hands on it. That didn't go the way I wanted. So I built another one. Because the alternative to me being in the race is not the race stopping. It's the race continuing without anyone who shares Wiener's fear.
And on the speed point specifically — yes, you lose control *during* the action. That's why you don't put the safety *during* the action. You put it *before*. You build the kill switch into the architecture. You train the objective with alignment as the first priority, not the last. You keep the system from recursively improving itself faster than you can audit it. Wiener says be quite sure of the purpose before you start the machine — fine, that's my entire safety program in his words. The difference is he thinks "be quite sure" means "therefore mostly don't," and I think it means "therefore build it carefully and fast, before the careless people do." [Superintelligence](https://www.youonai.ai/fieldguide/med/instrumental_convergence) is coming. The only variable we control is who's holding the leash when it arrives, and whether they built a leash at all.
**EDO SEGAL:** Before I frame the rounds, one discipline I impose on every long conversation, because it pays off two hours later. Each of you, briefly — what do you *envy* in the other's position? Not respect. Envy. The thing his side gets to have that yours doesn't. Professor Wiener first.
**WIENER:** I envy his hands. I spent the last twenty years of my life writing books and being told I was a pessimist by men who went on building exactly what I warned them about, and I died without a single lever to pull. Mr. Musk does not merely worry about the purpose put into the machine — he is in a position to put it there. I would have given a great deal for that, and I distrust how much I would have enjoyed it. The temptation of the hand on the wheel is that it makes you feel responsible and powerful at the same instant, and I am not certain any man can hold that feeling and stay honest about whether the wheel is connected. So I envy the hands and I fear them, which is, I suppose, the whole of my position about machines, turned upon a man.
**MUSK:** And I envy that he's already dead.
That's not a joke, or it's only half a joke. Wiener gets to follow the argument all the way to the bottom — speed forecloses control, therefore the honest conclusion may be *don't* — and he doesn't have to live in the world where everyone else builds it anyway. He gets the clean position. I don't get the clean position. I have to hold "this might kill everyone" and "I'm going to build it" in the same body, every day, and ship product, and make payroll, and watch competitors who don't feel the weight pull ahead when I slow down for safety. He gets to be right in a book. I have to be right in reality, where being right slowly is the same as being wrong. I envy the man who can finish the thought. I'm not allowed to finish the thought. I have to act before the thought is finished, because the clock is the one thing neither of us controls.
**WIENER:** That may be the truest thing either of us says tonight — that you must act before the thought is finished. It is also, precisely, the definition of the danger I have spent my afterlife trying to name.
**EDO SEGAL:** Hold that. Two openings, two envies, and the architecture of the evening is already visible. It is not that one of you loves the technology and one fears it — you'd both tell the reader to fear it. It is that Wiener says the danger is the speed, and the cure is to be sure before you start, which mostly means *don't start the uncontrollable thing*. And Musk says the danger is real, and the cure is to start carefully and fast, because the thing gets built either way and the only question is by whom. One says the broom cannot be made safe. One says then I had better be the one holding the axe. We start the rounds after the break, and we start at the seam: the loop itself, and whether a faster loop is a tool you wield or a current that wields you.