Who Watches the Watchman

**EDO SEGAL:** Professor Bentham, you keep reaching for the same fix, and I want to honor it by testing it to destruction. You say: the tower was always meant to be glass in both directions — the inspector inspectable, the public able to walk in, accountability flowing up as well as down. You insist the original sin of the digital version is that it kept the watching and threw away the reciprocity. So let me ask the builder's question back to you. In a world where the watchman is a machine-learning model whose own makers cannot fully explain its decisions — is your reciprocity even buildable? Can we watch *this* watchman?

**BENTHAM:** It is the right question and I will not pretend it is easy. In my prison, reciprocity was simple: cut a window, admit the magistrate, publish the accounts, let the public walk the gallery. The inspector was a man, and a man can be seen, questioned, and dismissed. Your watchman is not a man. It is, I am told, a function of such intricacy that its own engineers can describe what it was trained to do but not always *why* it did the particular thing it did. This is a genuine difficulty and I take it seriously. But I will not grant that it is fatal, and here is why. We do not, in fact, need to see *inside* the inspector's skull to hold him accountable. We never did. We held the human inspector accountable by his *conduct* and its *consequences* — by what he did, recorded, published, and judged against the welfare of the watched. The same discipline applies to the machine. Bind it by law to log its decisions. Compel disclosure of the objective it optimizes. Subject its outcomes to public audit — does it deny loans by race, does it sentence by poverty, does it farm attention into despair? You do not need to read the weights to read the harm. My reciprocity does not require transparency of the *mechanism*. It requires transparency of the *conduct and the consequence*. That, your century can build, and has refused to, and the refusal is a political choice, not a technical impossibility.

**EDO SEGAL:** So what you're saying, literally, is that the opacity of the model is a red herring — that we've never been able to see inside any inspector, human or machine, and we held the human ones accountable anyway by watching what they *did*. That's a strong move. Professor Zuboff, that sounds like exactly the [accountability you've demanded for forty years](https://www.youonai.ai/fieldguide/med/behavioral_surplus). Does Bentham's reciprocity satisfy you, or is there a reason it can't be built?

**ZUBOFF:** It's a strong move and I want to give him the half that's right before I take the half that isn't, because this is where I've spent my life and I won't be sloppy in it. He's right that we don't need to read the weights — that conduct and consequence are auditable, that the demand for mechanism-transparency is often a trap that lets companies say "it's too complex for you to understand" and walk away. Audit the outcomes. Yes. I've written it for decades.

Here's why his reciprocity, by itself, can't reach the harm I'm describing — and it's the precise point where surveillance capitalism is *new*, where it isn't just an old abuse at new scale. Bentham's accountability catches the *visible* harm: the loan denied by race, the sentence skewed by poverty. Those are real and we must catch them. But the core operation of the machine I study produces a harm that *leaves no victim who knows they were harmed*. When the system nudges ten million people's behavior two percent toward the outcome a client paid for, no single person can point to a moment and say *there, that's where my future was taken*. Each one experiences only their own choice, feeling free. The aggregate is steered; the individual feels sovereign. That's what I named [instrumentarian power](https://www.youonai.ai/fieldguide/med/instrumentarian_power) — not the old totalitarian power that breaks you so you submit, but a new power that *tunes* you so smoothly you never feel the hand. Bentham's audit asks "show me the injured party." Instrumentarian power's whole genius is that there *is* no injured party who can testify, because the injury is distributed across millions and disguised as their own freedom. You cannot subpoena a harm that every victim experiences as a preference. So his reciprocity is necessary and I'll fight for it beside him — and it is not *sufficient*, because the deepest extraction is engineered to be invisible even to the audit, even to the victim, even to the self.

**BENTHAM:** Now *that* is a serious objection, the most serious you have raised, and I will not wave it off. You are telling me there exists a class of harm that is real but, by construction, produces no testifying victim and no measurable injury to any individual — a harm visible only in the aggregate and only to the analyst. And you are right that my accountability, built for the visible deed, strains against it. But mark the cost of your own argument, because it is steep. You have just defined a harm that *cannot be pointed to* — that every supposed victim experiences as their own free preference, and that the analyst alone can see. Do you not feel the danger in that? You have constructed a wrong that the wronged deny they suffer, knowable only to the expert who insists they are deceived. That is the exact structure of every paternalism in history — *I* can see the chains you mistake for freedom, *I* will free you against your stated will. I spent my life fighting the priests who said precisely that about salvation. How is your instrumentarian harm different from a clergy's claim to see a corruption of the soul that the parishioner cannot feel?

**ZUBOFF:** Because I can show you the mechanism and the priest could only show you scripture, Jeremy — and that distinction is everything. The priest asks you to take the invisible corruption on faith. I'm pointing at a documented industrial process: the surplus extracted here, the model trained there, the prediction sold in that market, the intervention deployed at that timestamp, the behavior measurably shifted in the A/B test the company *ran itself and recorded*. The company knows it works — it pays billions on the knowledge that it works. I'm not asking you to believe in an unmeasurable soul-corruption. I'm asking you to read the company's *own* ledger, the one it keeps in private, where the manipulation is not a matter of faith but a line of revenue. The reason the victim can't testify isn't that the harm is mystical. It's that the harm was *engineered* to be sub-perceptual — the two percent nudge, below the threshold where you'd notice, summed across millions. The priest hid nothing because there was nothing. The company hides everything because there's everything. Your reciprocity is the right tool aimed at the visible deeds. I'm telling you the most profitable deed of all was specifically designed to fall just under the floor of what your audit can see — and that the design of invisibility is itself the proof of intent.

**EDO SEGAL:** Let me mark a second convergence, because it's important and it's fragile. You both agree — and this is news — that mechanism-transparency is a trap, that reading the weights is not the point, that conduct-and-consequence is the right axis of accountability. Convergence two. Where you split: Bentham says auditing visible conduct is sufficient; Zuboff says there's a sub-perceptual, aggregate-only harm — instrumentarian power — that's engineered to fall beneath any audit of individual conduct, and that its invisibility is the proof. And Bentham, you handed her the sharpest knife against herself — the paternalist's structure, the harm no victim admits — and she answered it with the company's own private ledger. The reader is going to have to weigh that exchange for a long time. We're at the end of the first hour. After the break, we leave the mechanism and go to the promise: does being watched actually make us *better*? Bentham staked everything on yes. Reform, or capture.