Reform or Capture

EDO SEGAL: Second hour. I want to open it personally, because the question of this round is the one I can't get out of my own skin. Professor Bentham, your entire defense of the tower rests on a single empirical bet: that being watched makes us better. That the prisoner reforms, the official decides more justly, the teenager who knows she's recorded posts more carefully — and that "more carefully" is an improvement. I have lived the counter-evidence. I built systems that watched people, and the people did not become better under the watching. They became more performed. More anxious. More like a curated version of themselves and less like themselves. So I have to ask you to defend the bet directly: when the watched modify themselves under the eye, are they reformed — or are they just captured, hollowed into compliance?

BENTHAM: You have asked the question on which my entire system stands or falls, and I will not retreat into theory to answer it. Let me grant the texture of what you saw, because I trust your eyes more than my abstractions. You watched people become performed. I believe you. But I must ask what they were before the watching, and against what you are measuring the loss. The man in the unwatched prison was not authentic and free; he was brutal and brutalized. The official deciding in the dark was not gloriously spontaneous; he was corrupt. When I say the watched improve, I do not mean they become more spontaneous. I mean their conduct improves — fewer cruelties, fewer thefts, fewer abuses — and I hold conduct to be what morality is finally about. You are mourning a loss of unselfconsciousness. I am counting a reduction in harm. These may both be real. The question is which weighs more, and I have always answered: the harm prevented weighs more than the spontaneity lost, because the spontaneity was a private luxury and the harm was someone else's broken body.

EDO SEGAL: So you're conceding the hollowing is real — the performed self, the lost spontaneity — and arguing it's a price worth paying because the conduct improves. That's an honest trade. Professor Zuboff, he's put it as a straight exchange: spontaneity for safety. Is that the right ledger?

ZUBOFF: It's an honest trade and it's the wrong ledger, and the reason cuts to the center of everything. Jeremy frames it as spontaneity-versus-safety, a private luxury weighed against a public good, and on those terms safety wins and he wins. But the thing being lost under the modern eye is not spontaneity. It's something with a function, something a democracy cannot survive without, and he's pricing it as a luxury because his calculus has no other slot for it. What's lost is what I call the right to the future tense — the capacity to face an open future and author it yourself, to be the kind of being whose next action is not yet decided, even by you. That isn't a private indulgence like unselfconsciousness. It's the precondition of moral life. A person who is continuously tuned toward an owner's objective, who experiences manufactured preferences as their own, has not lost a luxury. They've lost the thing that makes them a citizen rather than a subject — the capacity to will, to commit, to promise, to do otherwise. And here's the part that should chill the room: a population that has lost it doesn't notice, because the loss is engineered to feel like ordinary life. They still feel like they're choosing. The hollowing Edo saw isn't the surface symptom of a fair trade. It's the disease, and the disease is the slow death of the conditions under which self-government is possible. Twenty years ago, half the world lived in democracies. Now it's barely more than a quarter. I don't think that's a coincidence, and I don't think it's a luxury good that went missing.

BENTHAM: Madam, you have introduced a magnificent and dangerous phrase — the right to the future tense — and I must do to it what I did to all such phrases in my life: ask whether it is a real thing or nonsense upon stilts. I do not say that to belittle you. I say it because the cemetery of moral philosophy is full of "rights" that turned out to be feelings dressed in the costume of principle. So. Where does this right reside? Test it. You say a person continuously tuned toward another's objective has lost the capacity to author their own future. But Edo, raised by his parents, schooled by his teachers, shaped by his culture, advertised at his whole life — was he ever the unauthored author you posit? Were any of us? The self has always been written by forces it did not choose. Family, faith, fashion, fear. You speak as if there were once a pristine self-authorship that the machine has now stolen. I see no evidence it ever existed. The machine tunes us, yes — but so did the church, the school, the market, the mother. If the right to the future tense was already a fiction before the machine, the machine has stolen nothing.

ZUBOFF: And that — that right there — is the most dangerous sentence you've said all night, because it's the one the companies whisper to themselves to sleep. "We were always shaped, so what's the difference." Here's the difference, and it's not a feeling, it's a structure. My mother shaped me, yes — but my mother was visible to me, accountable to me, embedded in a relationship where I could resist her, argue with her, leave her, and where she bore the consequences of how she shaped me. The culture shaped me, but the culture was contestable — I could read against it, organize against it, vote against it. Every prior shaping force operated inside a web of mutual visibility and possible resistance. What's new — genuinely, structurally new — is a shaping force that is invisible, unaccountable, non-reciprocal, individually targeted, and optimized in real time against my own resistance. When I push back against the feed, the feed learns my pushback and routes around it. My mother never A/B-tested ten thousand variants of herself against me to find the one that defeats my will. The church couldn't update itself every millisecond to exploit the exact moment my defenses were down. You're right that we were always shaped, Jeremy. You're catastrophically wrong that this is the same kind of shaping. The difference between a parent and an optimizing adversary that studies your resistance to overcome it is the difference between being raised and being hunted.

EDO SEGAL: I'm going to do something I rarely do and stay silent for one more turn, because that needs to land. Professor Bentham — she just conceded your point and turned it into a blade. Yes, we were always shaped. But the prior shapers were visible, accountable, contestable, and slow. The new one is invisible, unaccountable, non-reciprocal, and faster than your resistance. Does the "we were always shaped" defense survive that distinction?

BENTHAM: It survives wounded, and I am too honest to claim otherwise. She has drawn a real line — visibility, accountability, contestability, speed — and a shaping force that fails all four is a different beast from a mother or a church, I grant it. Where I plant my flag is here: every one of those four properties is a property we could legislate back into the machine. Visibility — compel disclosure. Accountability — bind the objective by law. Contestability — grant the right to refuse, to leave, to be forgotten, to see and challenge the prediction. Speed — regulate the cadence, forbid the real-time exploitation of the moment of weakness. Each of her four objections names not an essence of the technology but a missing law. She says the machine is an optimizing adversary that routes around resistance. I say: then forbid it, by statute, from routing around resistance, and audit it for compliance. Her four-part distinction is the best argument I have heard tonight — and every part of it describes a tower with the windows torn out, which is to say, a building that can have its windows put back. I keep returning to the same conviction, battered but standing: the sin is in the missing reciprocity, and reciprocity is buildable.

ZUBOFF: And I keep answering: you cannot legislate the windows back into a building whose owners became the richest entities in human history precisely by tearing them out. The four properties you'd restore are the four properties they spent twenty years and trillions of dollars destroying, because the destruction is the profit. This isn't a building that lost its windows in a storm. It's a building designed without them, by people who understood that windows would end the business model. That's why I say abolish and you say reform — not because I love destruction, but because I've watched, for two decades, every reform get metabolized into a compliance department and a privacy policy nobody reads, while the extraction accelerated underneath. Your reciprocity is exactly the right dream. I'm telling you it's the dream they sell you to keep you reforming a thing that needs ending.

EDO SEGAL: Let me put one thing to you before we move, Professor Zuboff, because you keep saying the prior shapers were slow and this one is fast, and I want to make the speed concrete for the reader. You've written that the deepest battleground isn't behavior — it's attention. The thing the machine actually competes for, millisecond by millisecond, is where the human eye and mind land next.

ZUBOFF: Attention is the substrate of everything else, and it's where the asymmetry bites first. Before the machine can predict you, it has to capture you, and capture means winning the contest for your attention against your own intentions — your intention to sleep, to read the long thing, to sit with a hard feeling instead of swiping it away. There's an attentional ecology the way there's an ecology of a forest, and surveillance capitalism is an extractive industry operating inside it — clear-cutting the slow, the deep, the boring-but-nourishing forms of attention because the fast shallow kind is more predictable and therefore more sellable. Jeremy's tower kept order by being watched. This tower keeps order by farming the gaze, and a population whose attention has been strip-mined for a decade is not a population well-positioned to deliberate its way to self-government. You don't take a democracy by force anymore. You take its attention, and the rest follows.

BENTHAM: I will grant the ecology and deny the despair. An ecology can be protected; we have done it before, with rivers, with forests, with the labor of children. If attention is a commons being enclosed, then enclose the enclosers — by law, by public design, by the deliberate cultivation of slow attention the way a society cultivates parks. You describe a depletion. I hear a regulatory agenda. Every harm she names is real, and every one of them, to my ear, ends in a statute we have simply not yet had the will to pass.

EDO SEGAL: Hold that exact split, because it's the spine of the whole night — reform the tower or abolish it — and it's about to get its hardest test. Zuboff used a phrase she's been circling all evening, and it's the one from my own book that scares me most. The death cross — the moment the machine's prediction of you overtakes your authorship of you. Next round, we put the cross on the table.