Designing the Off Switch, or Designing the Person

EDO SEGAL: Let me get concrete, because we've been high for a while and I promised the kitchen table. Professor Skinner, in your analysis of these systems you keep returning to one missing piece: the extinction point. The off switch. You've said the reason people can't stop — the reason I couldn't stop at three in the morning — is that the machine has no built-in moment where the reinforcement ends, no signal that says the session is over. And your prescription is to design one in. Dr. Zuboff, you've named something that sounds adjacent but might be its opposite: the right to sanctuary, the right to disconnect, a protected space the instrument cannot enter. So I want to know if these are the same idea or rival ideas. Professor, the off switch first. What is it, and why is it the whole game?

SKINNER: It's the whole game because of a fact about reinforcement that the engineers building these systems do not know, because they never read me, because the culture decided I was distasteful and threw out the science with the distaste. Here's the fact. Every powerful reinforcement schedule in nature has come with an extinction point — a moment when the reward stops. Work ends when the day ends. Conversation ends when the other person tires and leaves. Even the slot machine ends when the money runs out. The organism stops because the environment stops reinforcing, and stopping gets the organism something — rest, the next activity, survival. The machine in this room is the first reinforcement schedule in the history of life with no extinction point. It never tires. It never runs out. It is available at three in the morning and it answers every single time. And so the behavior of stopping is never reinforced — stopping only removes the reward and produces nothing — and a behavior that is never reinforced does not occur. You didn't lack willpower, Mr. Segal. You were inside the only schedule ever built that gives the organism no reason to stop and no signal that it should. The off switch is the deliberate construction of the extinction point the machine forgot to include: a session limit, a temporal boundary, a slowing of the response as the hours pass — a designed moment where the reinforcement ends so the organism can do the most important thing an organism ever does, which is leave. It is the single most important piece of behavioral engineering this technology requires, and almost no one is building it, because the casino is reinforced for leaving it out.

EDO SEGAL: Dr. Zuboff — the off switch. Same as your sanctuary, or its enemy?

ZUBOFF: Adjacent, and then a chasm, and the chasm is the most important distinction I can draw tonight. Skinner's off switch is a feature designed into the machine by the people who own the machine. And I want to grant immediately that it's better than nothing — yes, build the session limit, yes, the extinction point, yes, I will sign Skinner's engineering memo this minute. But notice the structure of it. The off switch is given to you by the controller. You are still inside the box; the box has simply been designed to occasionally let you out, on a schedule the owner sets, for reasons the owner chose, revocable the moment the owner's incentives change. That is not sanctuary. That is parole. The right to sanctuary is something else entirely — it is not a feature of the machine, it is a limit on the machine, a space the instrument is forbidden to enter, established not by the owner's design but by the person's right and the law's force. The off switch says the casino will let you rest. Sanctuary says there are rooms of your life the casino may not enter at all — your home, your inner life, your child's developing mind, the attentional commons we all depend on. Skinner's off switch is benevolent design within the asymmetry. My sanctuary is the abolition of the asymmetry in certain protected zones. He's making the box more humane. I'm taking territory away from the box. And here's why the distinction is fatal and not pedantic: an off switch the owner can remove is not a right. It's a courtesy. And every courtesy surveillance capitalism has ever extended, it has withdrawn the moment the prediction got more valuable than the goodwill.

SKINNER: That's fair, and it sharpens rather than refutes me, so let me sharpen back. You're right that a feature the owner controls is parole, not freedom. But your sanctuary has the identical vulnerability you just accused my off switch of having — it depends on a power to enforce the limit, and that power is held by someone, and that someone can be captured exactly as the owner can. Your "law's force" is a contingency too, Shoshana — it's reinforced and extinguished like any behavior, by lobbying, by capture, by the slow erosion you've documented better than anyone. So we don't actually disagree about the kind of solution. We disagree about where to put the enforcement. You put it in law and rights — external limits on the box. I put it in design and contingency — the construction of the box and of the population. And the honest truth, which I'll say because the dead can afford honesty, is that we need both and neither is sufficient. Your sanctuary without my off switch is a right no one has the trained capacity to exercise — a door marked EXIT in a building where no one was ever reinforced for walking through it. My off switch without your sanctuary is a humane casino that stays a casino. The candle needs the law that forbids the box and the engineering that builds the capacity to leave it. You've spent your life on the first. I spent mine on the second. Tonight is the first time anyone's made me feel the lack of the half I skipped.

EDO SEGAL: I have to stop the room, because something just happened that I have been waiting all three hours for and did not expect to get. The reader can't see your faces, so let me mark it precisely: that was the first moment tonight where neither of you was arguing. You just assembled something together. Skinner: the off switch, the trained capacity to leave. Zuboff: the sanctuary, the protected zone law forbids the machine to enter. And you both said, in your own vocabularies, that each is useless without the other. The behaviorist needs the rights lawyer's limit; the rights lawyer needs the behaviorist's trained capacity. Number that — it's the largest convergence of the night, and it's the one I'd hand a legislator. Now let me complicate it before you get comfortable, because I'm not done being the referee. I'll declare my stake: I think this synthesis is right, and I think it's exactly what the software death cross makes economically impossible. The cost of building the casino is collapsing toward zero. The cost of building the sanctuary-plus-off-switch is not. So the market will flood with frictionless boxes and starve the protected rooms. Both of you just agreed on the cure. Tell me, honestly — does either of you believe it gets built?

ZUBOFF: No. Not on the current arrangement of power. And I want to be precise about my pessimism, because it is not despair — it is a diagnosis, and a diagnosis is the beginning of treatment, not the end of hope. The synthesis Skinner and I just built will not be built by the market, because the market is the casino and the casino is reinforced — there's his word, and it's the right one — for omitting exactly these features. It will be built, if at all, the way every limit on every extractive power in history has been built: by people who refuse, who organize, who name the theft, who make the political and legal cost of omitting the sanctuary higher than the profit of omitting it. It is built by countercontrol, exercised in the one venue Skinner's framework keeps wanting to dissolve into contingencies — democratic politics. The off switch and the sanctuary are not gifts the owners will design. They are territory the public will have to take. And whether the public still has the capacity to take it — after a decade inside the box that erodes that very capacity — is the open question of our age. That's why I'm furious, Edo. Not because I'm certain we'll lose. Because I can see exactly what we'd have to do to win, and I can see the instrument that's making it harder every day, and the clock is running in the casino's favor.

SKINNER: And I'll end my half on a note that will surprise the people who think I'm cold, because I think Dr. Zuboff and I have, in the last ten minutes, become co-authors of the same warning. She's right that the synthesis won't be built by the market. Where I differ is only in what I'd do about it, and even there we've converged more than I expected: she says organize and refuse; I say design the contingencies that make organizing and refusing more probable — which is the same instruction in two dialects. The thing I want the reader to carry is this. The off switch is not a small technical fix. It is the physical embodiment of the most important freedom there is — the freedom to stop, to leave the schedule, to do the unpredicted thing of walking away. A culture that cannot build an off switch into its most powerful instrument is a culture that has forgotten that stopping is a behavior, that behaviors are built, and that we are, right now, failing to build the one behavior on which all the others depend. Dr. Zuboff calls it sanctuary. I call it the extinction point. We are, at last, describing the same door. The tragedy would be a species that argued about the name of the door until it forgot how to walk through it.

EDO SEGAL: Hold there — both of you, at the same door, calling it by different names. Because the next round is the last one before I leave the room entirely. You've each spent a life imagining a whole world built on your principles. Skinner wrote his down as a novel. Zuboff, you wrote yours as the negative space inside an indictment — the world that wouldn't let surveillance capitalism happen. Two utopias. Let's visit them both, and find out whether they're the same place. After this.