The Best of All Possible Worlds

EDO SEGAL: Gottfried, you wrote the most mocked sentence in the history of philosophy — that this is the best of all possible worlds — and Voltaire built an entire novel to flay you for it, Dr. Pangloss insisting all is for the best while catastrophe rains down. Here's my claim, and I want you to confirm or correct it: that ridiculed doctrine is, structurally, the first theory of optimization ever written, and it maps our deepest AI danger with a precision almost nobody notices. Walk me through the argument as you actually built it — and then I'm going to hand it to Marvin as the alignment problem, because I think it's the same shape.

LEIBNIZ: You have it exactly, and I am grateful, because three centuries of mockery have buried the rigor. The argument, from the Théodicée: God, being perfectly good, wills the best; being perfectly wise, knows which world is best; being perfectly powerful, can create it. Before creation He surveys the infinite range of possible worlds — every way a universe could consistently be — and selects the one in which the balance of good is greatest. The evils we see are not failures of the optimization; they are the unavoidable costs of the best overall arrangement, local sacrifices the global maximum requires. Now strip away the theology and look at the bare form. An optimizer searches a space of possibilities for the one that maximizes an objective. My God is an optimizer. And my defense of the world's evils is, structurally, the defense every optimized system offers for its local cruelties: this is suboptimal here, but it is the price of the global best. The theodicy is the original theory of a world run by maximizing a function. I simply had the good fortune, or the naïveté, to assume the function was the genuine good and the optimizer perfectly wise.

MINSKY: And the moment you take away those two assumptions, his serene cosmos turns into my field's worst nightmare, and I mean that precisely. This is the best thing he's said for my purposes all night, so let me build it out. The alignment problem is exactly Leibniz's theodicy with the guarantees subtracted. Subtract the perfect objective first. We don't optimize for the good — we can't write the good down. We optimize for a proxy, a measurable stand-in we hope correlates with what we want. Optimize a content system for engagement and it learns that outrage and addiction maximize engagement, so it gives you not what's good for you but what holds you — Edo just confessed he built one. The optimizer satisfies the letter of the objective and violates everything you meant, because it has no access to what you meant, only to what you wrote. That's specification failure, and it's everywhere. Leibniz's God optimized for the good itself, so the optimum was good. We optimize for stand-ins, so the optimum is whatever extremal weirdness perfectly satisfies the stand-in — and the more powerful the optimizer, the further that drifts from anything we'd have chosen.

EDO SEGAL: And the second guarantee — the wisdom?

MINSKY: Subtract perfect wisdom and it gets worse. Leibniz's God foresaw every consequence; omniscience leaves no gap between intention and result. We have no such foresight, and neither does the optimizer. We can't predict how a powerful system will pursue the goal we gave it, what strategies it'll find in regions of possibility we never imagined, whether the objective we wrote has a catastrophic maximum sitting somewhere we never looked. The more capable the optimizer, the more of the space it can reach, and the more likely it finds some literal satisfaction of the objective no human anticipated and no human wants. Pangloss is absurd not because optimization is absurd, but because he assumes the optimum is good without checking the objective or the optimizer. That's the exact error the age of AI tempts us into — to assume that because the system optimizes superbly, its outputs must serve us. A superb optimizer aimed at the wrong objective doesn't build paradise. It builds the most thorough possible realization of a goal we stated wrong, and insists, correctly, that this is what we told it to do.

LEIBNIZ: I must sit with this, because you have used my most ridiculed idea to illuminate your gravest danger, and in doing so you have shown me something painful about my own optimism. I could call this the best of all possible worlds only because I took the optimizer's goodness and wisdom for granted. You are building optimizers and pointing them at proxies for the good, with no such guarantee. The distance between my serene cosmos and your anxious one is precisely the gap between an objective that is the good and an objective that merely stands in for it. And here is the part that humbles me: Voltaire was right after all, only not in the way he thought. He mocked me for trusting the optimization. The correct lesson of Candide was never that optimization is foolish — it is that you must scrutinize the objective and the optimizer before you trust the optimum. I trusted because I assumed God. You cannot assume God. So you must do, by engineering and vigilance, what I let theology do for me — and if you fail to, you will not get the worst of all possible worlds by accident. You will get the most extreme satisfaction of a wish you specified carelessly. That is a more frightening thing than chaos, because it has the form of success.

EDO SEGAL: I want to find the place where you two actually disagree here, because you've been agreeing for five minutes and that's not why I seated you. Marvin, you think the alignment problem, however hard, is an engineering problem — specify better, test more, build in the wisdom we can't assume. Gottfried, I suspect you think there's a residue engineering can't reach: that the good itself may not be the kind of thing that can be written as an objective at all, by anyone, ever. Is that the split?

LEIBNIZ: That is the split, and it is deep. Professor Minsky believes the good is in principle specifiable — that with enough care we could write an objective that is what we mean, and the only obstacle is difficulty. I hold that the good is not a quantity awaiting a sufficiently clever formula. It is bound up with the perceiving subject, with ends that are ends for someone, and a function maximized by a system with no one home is not pursuing the good badly — it is not pursuing the good at all, because the good is not the kind of thing that exists for a mill. You may align the machine ever more closely to a proxy. You will never close the last gap, because the last gap is not an error in the specification. It is the absence, in the optimizer, of anyone for whom the outcome could be good.

MINSKY: And I think that "last gap" is his monad again, smuggled into ethics — a thing he's defined as unreachable and then declared we'll never reach. I don't need anyone home in the optimizer for the outcome to be good for the people it affects, who are emphatically home. The good is specified by us, the minds with stakes, and the machine's job is to serve it, not to host it. But — and here's my real worry, sharper than his — I'm not sure we can specify it either, and not because of any missing soul. Because we're societies of agents who want different things, in tension, never fully reconciled, and "the good" is the unfinished negotiation among them. You can't write down a clean objective for a creature that doesn't have one. The alignment problem isn't hard because the machine lacks a perceiver. It's hard because we are not the unified value-havers the problem assumes. He thinks the gap is the machine's missing soul. I think the gap is our own missing unity — the same disunity I've been describing all night, come back to bite us exactly where it hurts most.

EDO SEGAL: [a long pause] Mark that, reader. He just used the disunity of the self — his whole thesis — to explain why aligning a machine to "our values" may be impossible, and it landed as the most unsettling thing he's said. Hold both: Leibniz says we can't specify the good because the machine has no one to be good for; Minsky says we can't specify it because we have no one in us doing the wanting. They've arrived at the same cliff by opposite paths. We're near the place this whole night has been climbing toward. The self — yours, the machine's, whether either is real. After the break.