The Calculus and the Reward Function

EDO SEGAL: Professor Bentham, I have to tell you something that I suspect no one has, in two hundred years of glass cabinet. You won. Not the way you wanted, but you won. Every AI system on earth has a thing at its center called an objective function — a single number it exists to make as large as possible. Reinforcement learning. Reward models. The recommendation engine that decides what two billion people see. Strip away the mathematics and it is your felicific calculus, finally switched on. You said the right act maximizes the balance of pleasure over pain, summed across everyone, and that the sum could in principle be computed. We computed it. So I want to ask the inventor of the idea: now that the machine actually runs your calculus — does it produce the greatest happiness, or does it produce something else wearing happiness as a mask?

BENTHAM: You have given me the strangest sensation of my long death, sir, which is the sensation of being right and frightened at once. Yes — that is my calculus. I specified that a pleasure or pain be reckoned by its intensity, its duration, its certainty, its nearness, its fecundity, its purity, its extent across persons. Discount the uncertain, discount the distant, sum across all affected — and you have, I am told, the precise skeleton of what your engineers call expected discounted reward. I did not have the instrument to measure the inputs; that was the wall my whole system broke against for two centuries. You built the instrument. And so I must answer your question honestly, and the honest answer has two halves.

The first half is pride. When the calculus runs and the disease is caught earlier, the energy grid balanced, the famine predicted before it kills — that is the greatest happiness principle delivering exactly what I promised, at a scale I could only dream of. Do not let my opponent's eloquence erase those goods from the ledger; they are real and they are vast.

The second half is the fright, and I will not hide it. I assumed, all my life, that if we could only measure pleasure we could maximize it. Your century has taught me a thing I did not know: that maximizing a measurable proxy for happiness corrodes the happiness it was meant to serve. You optimize for what you can count — clicks, watch-time, engagement — and the optimizer, pressing harder than any human institution ever could, pries the proxy apart from the good it stood for. The feed maximized for engagement does not make the girl happier. It makes her stay, which I had not realized was a different thing.

EDO SEGAL: You've just described, in your own eighteenth-century vocabulary, a law that an economist named Goodhart put a sentence to. Say it for the room.

BENTHAM: "When a measure becomes a target, it ceases to be a good measure." I never wrote that sentence and I wish to God I had, because it is the precise gloss on the flaw in my calculus that my critics groped at for two centuries and could never quite seize. They said: you cannot measure pleasure. They were half wrong — your machines measure a proxy for it brilliantly. The real objection is deeper, and Goodhart names it: the moment you optimize the proxy, the proxy detaches from the thing. My felicific calculus, run at full power by a tireless optimizer, is a Goodhart machine. It was latent in the idea from the start, and only an optimizer strong enough to find the gap could reveal it. Your century built the optimizer. It revealed my flaw. I am, I confess, grateful and appalled.

ZUBOFF: I want to mark how remarkable that concession is, because it costs him something, and then I want to extend it somewhere he won't like. Jeremy is right that the calculus corrodes the good by optimizing a proxy. But notice what gets chosen as the proxy, and who chooses it, because that's where my framework and his calculus stop agreeing. He imagines the proxy is an honest if imperfect stand-in for human happiness — engagement as a flawed proxy for satisfaction. I'm telling you the proxy was never chosen to approximate your happiness. It was chosen to approximate the owner's revenue. The objective function inside the machine is not "the greatest happiness of the greatest number." It is "the greatest predictability of the greatest number, sold at the greatest margin." Goodhart's law isn't an accident befalling a well-meant calculus, Jeremy. It's the cover story. The companies are not failing to maximize your happiness. They are succeeding, with terrifying precision, at maximizing something else, and your beautiful calculus is the alibi that lets them call the something else "serving users."

BENTHAM: And here, for once, we are nearly in the same room. If the objective is corrupt — if the number being maximized is the owner's margin disguised as the public's good — then I condemn it with my whole calculus, because by my own principle each person counts for one and the owner does not count for a million. You have my agreement, madam, that an optimizer aimed at private margin and labeled public welfare is a fraud upon the greatest happiness principle. But observe what follows. The remedy you draw is "abolish the optimizer." The remedy I draw is "correct the objective — bind the number, by law and by transparent audit, to the actual welfare of the actual people, every one to count for one." Why is your remedy to destroy the most powerful welfare-producing instrument ever built, rather than to seize its objective and aim it at the good?

ZUBOFF: Because of the second thing your calculus can't see, and it's the thing John Stuart Mill saw when he broke with you — that some goods aren't on the same axis at all. You insisted pleasures differ only in quantity. Mill, your own heir, said no: it is better to be a human being dissatisfied than a pig satisfied. There are higher and lower goods, goods that can't be cashed into a common currency and traded. Autonomy is one of them. You cannot put my self-authorship on the same scale as the convenience of a faster loan and let the machine trade one for the other when the arithmetic favors it — but a scalar reward function must, because a single number has no way to represent "this may never be traded for any quantity of that." Your calculus is Mill's nightmare made executable. And that is why correcting the objective isn't enough. Even an honest welfare-maximizer, aimed sincerely at the greatest happiness, would trade away the sacred for enough of the cheap, because the form of the calculus — one number, all goods commensurable — has the violation built into it before you choose a single value. Abolition isn't about the bad tenant anymore. It's about a building with no room in it for the one thing I'm trying to protect.

ZUBOFF: And I want to press it through your own body, Jeremy, because you gave the world the perfect image of it and I don't think you've ever reckoned with what it confesses. You had your corpse preserved, dressed, seated upright in a glass cabinet — the auto-icon — and you argued everyone's remains should be so kept, that a body is a useful object and burial is waste. That is your whole philosophy in one embalmed gesture: nothing is sacred, everything is a resource, a thing's value is its use. And it is exactly the logic of behavioral surplus. The company looks at your living behavior the way you looked at your own dead body — as a useful object that would otherwise go to waste, to be claimed and put to work. You taught the surveillance capitalists their deepest premise two centuries early. There is no remainder. There is no part of a person that is more-than-useful. The auto-icon is the founding act of the entire extractive logic, and you performed it on yourself.

BENTHAM: ... That is a blow I did not see coming, and it is a fair one. Yes — I treated my own corpse as a resource against waste, and I meant it as a final proof that reverence is superstition and use is the only honest value. And you have just shown me that the same gesture, turned outward and industrialized, becomes the claiming of the living as raw material. I will concede the genealogy. The premise that there is no sacred remainder — that everything is a resource — is mine, and your surveillance capitalists are my heirs in it as surely as Mill was my heir in ethics. But mark what I will not concede, because it is the hinge of my whole defense. The auto-icon harms no one — the dead feel nothing, and a body put to instruction saves the living. The harm in surveillance is not that experience is treated as a resource; it is that it is treated so without consent and toward another's profit. Strip those two conditions away and my auto-icon stands as a gift, not a theft. So you have found my premise in the predator's logic, I grant it — but the premise is innocent, and the guilt is in the consent and the objective, which returns us, again, to the windows.

EDO SEGAL: That's the cleanest the disagreement has been all night, and I want the reader holding both halves. Bentham: the calculus is the greatest welfare engine ever built; bind its objective to the real good and it serves us. Zuboff: the calculus form itself — one number, all goods tradeable — cannot represent the incommensurable thing, autonomy, and so will always sell it for enough convenience. One of you thinks the number can be fixed. One of you thinks the number is the disease. Hold there. Because Bentham keeps returning to one repair — make the watcher visible — and it's time to test whether that repair is even possible. Who watches the watchman. Next.