The Surplus and the Self

EDO SEGAL: Professor Zuboff, you gave the culture a word that broke something open in me when I read it — not "surveillance capitalism," the famous one, but a quieter one underneath it. Behavioral surplus. I want you to tell it the way you'd tell a parent at a kitchen table who has never heard the phrase and is about to hand her kid a tablet. What is the surplus? And then, Professor Bentham, I'm going to ask you to do the hard thing — to steelman it before you attack it.

ZUBOFF: Sit at that kitchen table with me. Your daughter opens an app. She's using it to talk to her friends — that's the service, that's the deal she thinks she made. But while she does, the app is recording something she never agreed to give and doesn't know she's giving: not the content of her messages only, but the shape of her behavior. How long she hovered before she replied. Which photo she paused on. What time of night her mood drops. The order in which her attention moves. None of that is necessary to deliver the service she asked for. It's extra — surplus to the transaction — and it gets claimed, automatically, because the company owns the infrastructure her life now runs through. That surplus is the most valuable thing in the building, more valuable than the service, because when you pour enough of it into machine intelligence, the machine learns to predict her — and a prediction of a sixteen-year-old's behavior is something a thousand companies will pay for. She thinks she's the customer. She's the raw material. The product is the prediction of her, and she will never see it, never be paid for it, never be allowed to say no to it, because the taking was designed to be invisible. That's behavioral surplus. It is the quiet center of the whole machine.

EDO SEGAL: Professor Bentham. Steelman first. Tell me what she gets right before you tell me what she gets wrong.

BENTHAM: I can do that honestly, because she has seen something true. What she gets right is that there is an asymmetry of knowledge so vast it amounts to an asymmetry of power, and that this asymmetry is undisclosed. The girl does not know what she is giving. The company knows exactly what it is taking. And undisclosed asymmetry is precisely the condition I warned of when I insisted the inspector must himself be inspectable — power exercised in secret is power abused. She is right that the secrecy is the crime, and right that the watcher's hidden interest changes the moral character of the watching. That is a real contribution and I salute it. There is the steelman.

Now the two places it fails, and they are load-bearing. First — and she will hate this — the word "theft" smuggles in a conclusion that the argument has not earned. Theft requires that a thing of value be taken from its owner, leaving the owner poorer. But the girl's hesitation, her pause on the photograph — these were not in her possession as property. They were behavior emitted into a shared situation, and the record of them costs her nothing she had. She has the same messages, the same friends, the same evening. The surplus is a new thing, created by the observation, that did not exist before the watching and takes nothing from her store. You may dislike that someone profits from it. But "I dislike that you profited from observing me" is a very different and much weaker claim than "you stole from me," and the rhetorical slide between them is doing all her work. Second: she keeps describing the harm as the taking, in the abstract, with no ledger. I keep asking — what is the injury? Show me the girl made worse off. If the prediction is used to recommend a song she loves, where is the harm? If it's used to manipulate her into despair, there is the harm — and notice it lives in the use, in a downstream consequence I can measure, not in the mere fact of observation. Convict the harmful use and I am entirely with you. But she wants to convict the watching itself, and the watching itself, absent a harmful use, is a tree falling where no one is hurt.

ZUBOFF: No. And this is the deepest disagreement we'll reach tonight, so let me be exact, because Jeremy has just made the single most seductive error in the entire debate. He says the surplus costs me nothing I had — that the watching takes nothing, that the harm only arrives in the downstream use. That is precisely the logic that has anesthetized two billion people. Here is what it misses. The capacity to predict me at scale is itself the injury, before any particular use, because it transfers to someone else a power over my future that was mine. You asked for the ledger, Jeremy — here is the entry you keep refusing to write down. On the asset side of the company's balance sheet sits a thing called certainty about what Shoshana will do. That certainty did not appear from nowhere. It was subtracted from me — from the open, undetermined quality of my own next action, which is the very substance of what it means to be a free person. You can't have it both ways. Either the prediction is worthless, in which case why is it the most valuable asset class in the history of capitalism — or it's worth something, in which case the something was taken from the one place it could have come from, which is the future I was supposed to author. The death cross you'll hear about later isn't only economic. It's the moment the machine's certainty about me crosses over and exceeds my own — and on that day, whatever it does next, I have already been demoted from author to subject.

BENTHAM: But this is mysticism dressed as economics, and I must press it. You say certainty about your behavior was "subtracted from you." Madam, the certainty was manufactured by the observer out of regularities you produced — it was not sitting in you like a coin in a purse. When an astronomer predicts the comet, he does not steal the comet's freedom; the comet had none to begin with, and his prediction subtracts nothing from its orbit. Your argument requires that there was a free, unpredictable future "in" you that the prediction consumed. But either your behavior is regular enough to be predicted — in which case the regularity was always there, and the observer merely found it — or it is not, in which case the prediction fails and there is no harm. You cannot claim simultaneously that the machine predicts you accurately and that the accurate prediction destroyed a freedom that, had it existed, would have made the prediction impossible.

ZUBOFF: That's a beautiful trap and I'm going to walk straight out of it, because it rests on confusing a comet with a person. The comet doesn't read the prediction. I do — or rather, the system acts on me with the prediction, and that is the entire difference your eighteenth-century determinism can't see. The astronomer predicts the comet and leaves it alone. The surveillance system predicts me in order to intervene — to tune the feed, time the notification, set the price, arrange the stairs, as Edo said, so that tomorrow's behavior falls where the owner wants. It doesn't find my regularity. It engineers it, then sells the engineered certainty as if it had merely discovered a natural law. You, of all people, should see this — it's your own Panopticon mechanism. The prisoner who knows he might be watched modifies himself. You called that reform. I'm telling you that when the watcher has a hidden commercial objective and the power to act on the prediction, the same mechanism becomes the manufacture of the human in someone else's image. The freedom isn't a coin in a purse, Jeremy. It's the difference between a future I write and a future written for me and handed back disguised as my own choice.

EDO SEGAL: I have to stop us here for one beat, because something just happened that the reader needs marked. Bentham, you said the comet has no freedom for the prediction to steal. Zuboff, you said the difference is that the system acts back on the predicted. And underneath both of you is the oldest question there is — whether a human being is the kind of thing whose future is open in a way a comet's isn't. That question is bigger than this round and we are going to come back to it on a higher floor. But before we leave: Professor Bentham, the mechanism she just described — predict, then intervene to fulfill the prediction — is your mechanism. Self-regulation under observation. Do you recognize your tower in her account of the harm?

BENTHAM: I recognize it, and the recognition is uncomfortable, which is how I know we have reached something real. Yes — my prisoner modifies himself under the eye, and I called it reform because the modification ran toward an end I held to be good: lawfulness, industry, health. She is telling me that when the end is instead the watcher's profit, the identical mechanism produces not a reformed man but a farmed one. And I will concede the structure of her point entirely. The mechanism is morally neutral; it takes its color from the objective it serves. Where I will not yet yield is the conclusion. She says: therefore abolish the mechanism. I say: therefore fix the objective. The eye that reforms toward the public good and the eye that farms toward private profit are the same eye pointed at different ends, and the whole of statesmanship is the discipline of pointing it rightly. Two centuries ago I would have said this with total confidence. Tonight I say it with somewhat less, which she may take as a victory.

EDO SEGAL: Before I number that, I want to give you a piece of your own earlier work, Professor Zuboff, because it complicates the "it's all theft" picture and you'd be the first to insist on the complication. Forty years ago, before surveillance capitalism, you drew a distinction in a paper mill that the whole field still runs on — between automating and informating. The same machine that displaces a worker's skill also generates new knowledge that could deepen human understanding. Two effects, one event. Doesn't that distinction cut against your own indictment? Isn't the surplus also informating — also a genuine gift?

ZUBOFF: It does, and I will never deny it, and you're right that I'd be a fraud if I did. That distinction is the foundation under everything I've built. Every smart machine both automates and informates — it displaces an old way of knowing and it produces new data, new reach, new capability. When AI lets a backend engineer build a whole interface, or a kid in Lagos write production code, that's informating; that's the democratization of capability and it is real and it is good. I have never been against the dividend. Here is what forty years taught me and what makes me hard where the optimists are soft: the informating dividend does not distribute itself. In the paper mills, the new knowledge was real — and it flowed almost entirely upward, to the managers and engineers who already had the authority to use it, while the floor workers who'd lost their embodied knowing got thin monitoring jobs. The potential was democratic. The institutions made it oligarchic. So when I say "theft," I'm not denying the gift. I'm saying the gift and the extraction are the same event, and that without institutions built to capture the dividend for the people who generate it, the default — every single time in my evidence — is automation without informating, dividend captured by the few, surplus extracted from the many. The optimist points at the gift. I point at the distribution, because the distribution is where the human outcome actually lives.

BENTHAM: And here, madam, you have walked straight into my house and sat down, whether you meant to or not. You have just said the harm is not in the machine but in the institutions that fail to distribute its dividend — that the same surplus, under the right institutional design, flows to the many instead of the few. That is not a case for abolition. That is a case for institutional reform — for building, by collective will, the structures that aim the dividend rightly. The eight-hour day, the public park, the labor law: each was a dam built in the current of a market that would otherwise have taken everything. You believe in dams. You've spent your life describing the ones we failed to build. I am simply the man insisting that the answer to a missing dam is to build the dam, not to drain the river.

ZUBOFF: Except the dams I'm calling for are exactly the ones the owners spent twenty years and trillions of dollars ensuring could not be built — and that's the difference between your century and mine, Jeremy. In your time the factory owner could resist the eight-hour day but could not predict and preempt the organizing that won it. The owner of the modern tower can. He sees the dam coming before the builders do, because he's watching the builders, and he routes around it. But I'll grant your point its due weight: yes, my deepest position is institutional, not technophobic. I want dams. I've just lost faith that you can build a dam in a river whose every drop is owned and surveilled by the party you're trying to dam.

EDO SEGAL: Now I'll number the convergence, and it's bigger than I first thought. Convergence one: you both agree the watching mechanism is morally neutral, that everything turns on the objective and the accountability — and you both agree the real battleground is institutional, that the dividend is real but doesn't distribute itself, that the answer is a dam built by collective will. You split, hard, on whether the dam can be built at all in a surveilled river. Hold that. Because the objective Bentham wants to fix has a precise shape, and it's the same shape as the thing inside every AI: a number to be maximized. The calculus, and what happens when you finally build it. Next round.