Shoshana Zuboff vs B F Skinner on AI · Ch9. The Candle and the Countercontrol ← Ch8 Ch10 →
Txt Low Med High
HOUR THREE — THE CANDLE, THE OFF SWITCH, THE UTOPIAS
Chapter 9

The Candle and the Countercontrol

Page 1 · The Candle and the

**EDO SEGAL:** There's a metaphor at the center of my book that I need both of you to take a position on, because I think it's where you'll either finally agree or finally break. I called it the candle. The idea is that as the machine automates more and more of what we do, there's a flame it can't reach — some core of judgment, taste, care, the thing only a person can hold — and the [candle](https://www.youonai.ai/fieldguide/med/candle_in_the_darkness) is what's left when everything automatable has burned away. The optimist's hope is that the candle is real and inextinguishable. The pessimist's fear is that there is no candle — just a series of things we hadn't gotten to yet. Dr. Zuboff, Professor Skinner has spent his life arguing there's no special flame, that it's contingencies all the way down. So I'll ask you first. Is the candle real? And if it is, what is it made of?

· · ·
Page 2 · The Candle and the

**ZUBOFF:** The candle is real, and I can tell you exactly what it's made of, because I've spent forty years watching people try to extinguish it and watching it refuse. It is made of *countercontrol* — the capacity of the person who is acted upon to act back. Skinner himself named it; he just underestimated it. Every system of control in history has produced its own resistance: the worker who slows the line, the citizen who organizes, the consumer who boycotts, the child who says no to the mother. The candle is not a mystical inner soul floating free of causation. It is the *living fact that the controlled are also controllers* — that the loop, in a human world, always runs both ways, because a person is not only an effect of their environment, they are also a cause acting *on* it. The flame is the future tense: the capacity to do the thing your history did not predict, and in doing it, to *change the history that's acting on the next person.* That is what the instrument is built to extinguish, because countercontrol is *noise* in a prediction — and a prediction with no noise is a person with no candle. So is the candle real? Yes. Is it inextinguishable? No. That's the whole terror. It is the most extinguishable thing there is, and it has never before faced an instrument designed, owned, and funded specifically to put it out.

**EDO SEGAL:** Professor Skinner. She just built the candle out of *your* concept — countercontrol — and dared you to call it nothing. Do you?

· · ·
Page 3 · The Candle and the

**SKINNER:** I don't call it nothing. I call it *real and badly described*, and the difference matters because it determines whether we can protect it. Dr. Zuboff is right that countercontrol is real, that I named it, and that it's the heart of the matter. She's right that it runs both ways and that a person both is shaped by and shapes their world. Where I correct her — and it's the correction that could save the candle instead of merely mourning it — is here: countercontrol is *not* a flame that either exists or doesn't, lit by some inner spark. Countercontrol is a *behavior*, and like every behavior it is *built by contingencies* — which means it can be strengthened or weakened, reinforced or extinguished, by design. A population's capacity to resist is not a constant. It is a *trained repertoire.* The worker who slows the line learned to; the citizen who organizes was shaped to; the child who says no was reinforced for refusal somewhere in her history, or she'd never have produced it. So here is the consequence Dr. Zuboff's beautiful, fatalistic candle can't reach, and it's the most hopeful thing I'll say tonight: *if countercontrol is a behavior, we can deliberately build more of it.* We can arrange environments — schools, communities, even the machine itself — that reinforce refusal, inspection, deliberation, the saying of no. She treats the candle as something we can only *defend* and *watch die.* I treat it as something we can *manufacture more of, on purpose, at scale.* Her candle flickers in the wind and we pray. Mine is a fire we can teach the whole species to light.

· · ·
Page 4 · The Candle and the

**ZUBOFF:** And that is the most generous and the most dangerous thing you've said, because it contains the trapdoor I keep falling toward and keep catching myself on. "We can manufacture more countercontrol." *Who is "we," Skinner?* The instant you make resistance a *manufactured* behavior, you've put it under the control of whoever runs the manufacturing — and now the resistance the people produce is the resistance *the manufacturer selected for*, which means it's no longer resistance against the manufacturer. You've built a population that says no to everything *except the hand that taught it to say no.* That's not countercontrol. That's the most elegant control ever devised — control that has captured even the appearance of its own opposition. The candle has to be the thing that *wasn't* lit by the controller, or it isn't the candle at all. The genuine no is the no the system *didn't* engineer.

**SKINNER:** Then you've made the candle *uncausable*, and an uncausable thing is a ghost. If the only genuine no is the no that no contingency produced, you've defined freedom as an effect without a cause — which is exactly the pre-scientific superstition I spent my life trying to bury, dressed in new clothes. *Every* no has a cause, Shoshana. The slave's no, the citizen's no, your no, my no — every one of them was built by a history. You can't rescue the candle by making it acausal; you can only rescue it by making it *built by contingencies the controller doesn't own.* The mother's no is genuine even though she taught it, because *she* taught it, not the platform. The teacher's no is genuine even though it was trained, because the teacher's reward was the student's freedom. The candle is real, and it's caused, and the only question — the question under every question tonight — is *whose contingencies built it.* Not whether it was built. It was always built. The fight is over the builder.

· · ·
Page 5 · The Candle and the

**EDO SEGAL:** And there it is — the whole night in one exchange, so let me mark it as the deepest convergence of all, because it doesn't look like agreement and it is. You *both* believe the candle is real. You *both* believe it's the thing that matters. You agree it's countercontrol — the capacity to say no. And you've narrowed your entire disagreement to a single question: whether a *no* that was built by someone's contingencies can still be a real *no.* Skinner: yes, if the builder is the right one. Zuboff: no, the real *no* is precisely the one the builder failed to build. That is a genuine, irreducible philosophical fork — and it is the fork the reader has to cross alone, because the two best people on earth just crossed it in opposite directions in front of you. Hold it. Because the next round makes it concrete: if the candle is countercontrol, then the most important feature of any of these machines is whether it lets you put it down. The off switch. Or the sanctuary. After this.

· · ·
Continue · Chapter 10
Designing the Off Switch, or Designing the Person
← Prev 0%
Ch9 Next →