Opening Positions

BENTHAM: Thank you. I shall begin where all clear thinking begins — with the rejection of mystification. My opponent will speak tonight of theft, of dignity, of the violated inner self, and these words will move you, because they are designed to move you. I ask you to notice that not one of them is a quantity. I built my entire life's work on the conviction that morality must be taken out of the hands of the priests and the poets and given to the accountants — that the only honest question about any law, any machine, any tower, is this: does it produce more pleasure than pain, summed across every person it touches, each to count for one and none for more than one? That is the greatest happiness principle, and it is the most democratic idea ever conceived, because it refuses to let the king's joy outweigh the pauper's agony.

Now apply it to the watchtower. Why did I design the Panopticon? Not to oppress — I despised oppression, I fought it everywhere I found it. I designed it because the prisons of my day were schools of cruelty and disease, the punishments arbitrary and savage, and I saw that if a man believed himself observable at any moment, he would regulate his own conduct without the lash. The genius of the design was that the inspector need not even be present. The mere architecture of possible observation transfers the burden of control from the guard to the prisoner, who becomes the principle of his own good behavior. No chains. No beatings. Reform achieved through the gentlest possible means — the knowledge that one might be seen.

I will tell you what I could never solve, and what your century has solved for me. My tower had a scaling problem. One watchman cannot truly attend to a thousand cells; the threat was a bluff, and the inmate's uncertainty rested entirely on the inspector's limitation. Your machine intelligence removes the limitation. It can in fact watch every cell at once, tireless, forgetting nothing, its capability rising with every doubling of scale. The bluff is no longer a bluff. And so I say to you plainly: the digital age has not perverted my dream. It has completed it. For the first time in history, a society can be made fully legible to itself, and what is fully visible can at last be fully governed toward the good. The teenager who knows her searches are recorded posts more carefully. The official who knows his decisions are logged decides more justly. Visibility is the friend of the vulnerable — darkness is where cruelty has always sheltered. My opponent wants to put out the light and call it freedom. I want to finish wiring the building and call it reform.

EDO SEGAL: Shoshana.

ZUBOFF: That was a brilliant opening, and I agree with the first half of it completely, which is going to surprise both of you. Jeremy Bentham is right that indignation is not an argument, and he is right that darkness shelters cruelty, and he is right that visibility, made accountable, is the friend of the vulnerable. I have spent forty years demanding visibility — of the powerful, of the corporations, of the mechanisms. So let me tell you exactly where the tower he is so proud of becomes the thing I have devoted my life to fighting, because the seam is precise and the whole evening lives inside it.

His Panopticon had one feature he keeps mentioning and one feature he keeps omitting. He mentions that the prisoner is watched. He omits that in his original design, the inspector was also watched — the public could enter, the institution was meant to be glass in both directions, accountability flowing up as well as down. Strip out that reciprocity and you do not have Bentham's tower. You have its corpse. And that is exactly what was built. We are watched by systems we cannot watch back, owned by entities we cannot inspect, optimized toward objectives we are never shown. The reciprocity is gone. What remains is pure asymmetry, and asymmetry of vision is asymmetry of power.

But here is the part that is genuinely new — newer than Bentham, newer than the prison, and this is the work of my life. The old surveillance watched what you did. This watches what you do, and then it does something the watchman in the tower never dreamed of: it claims your behavior as free raw material. I call it behavioral surplus — the residue of your experience that you generate just by living, that nobody asked you to generate, that gets taken because the infrastructure of your life is owned by someone else. The clicks, the pauses, the heartbeats, the rhythm of your hesitation. It flows into what I call the factory, which is what we politely call machine intelligence, and out the other end come prediction products — computations of what you will do next — sold in markets that trade in your future. The point was never to watch you, Jeremy. The point is to predict you, and prediction, at scale, becomes the power to modify you. The tower doesn't just keep order. It manufactures the order it wants, in you, and sells the certainty to a client. That is not reform. It is a coup — a coup from above, against the human right to author your own future. And the reason your century calls it convenience instead of conquest is that, unlike you, it had the sense to hide the tower. You were too honest to hide it. They learned from your honesty exactly what to conceal.

EDO SEGAL: Before I open the rounds, one discipline I impose on every long conversation, because it pays off two hours later. Each of you, briefly — what do you envy in the other's position? Not respect. Envy. The thing their side gets to have that yours doesn't. Professor Bentham first.

BENTHAM: I envy her certainty about the sacred. I genuinely do. She can stand in front of the tower and say this violates something in the person that must not be violated, and the whole room nods, and she never has to say what the something is or prove it weighs more than the benefits the tower delivers. My system gives me no such luxury. I have to count. I have to admit that the loan was granted faster, the disease caught earlier, the fraud detected, the lost child found — all real pleasures, all on the ledger — and then argue that the harm still outweighs them. She gets to invoke a thing called dignity that, by her own account, cannot be measured, and use it to trump every measurable good. I spent my life calling that move "nonsense upon stilts." I will confess, across two centuries, that the stilts are taller and steadier than I admitted.

ZUBOFF: And I envy his clarity of conscience. Jeremy Bentham believed, all the way down, that he was helping — that the tower was love. He never had to live inside the doubt I live in. Because I am describing a system that did find your lost child, that did catch the disease, and I have to tell people that the machine which delivered those gifts is also dismantling the conditions of their freedom, and I have to ask them to give up some of the gifts to keep the freedom — and I watch their faces close, because the gifts are real and the freedom is abstract until it's gone. He got to be the optimist. I am the one standing at the door of the comfortable house saying the foundations are being sold out from under you while you sleep. I envy a man who could believe the tower was kindness. I lost that ability somewhere in the second decade of the evidence.

BENTHAM: That may be the truest thing either of us says tonight.

EDO SEGAL: Hold that — both of you just located the danger in opposite places, and the reader should see the architecture of the whole evening in it. Bentham says the danger is that we'll fear the tower so much we put out the light that keeps us safe. Zuboff says the danger is that we'll love the tower so much we never notice it has stopped serving us and started farming us. Two people who both want a good and visible society, who cannot agree on whether the watching that produces it is a gift or a theft. We start the rounds with the building itself. The Panopticon — Bentham's act of love. After the break.