The Best of All Possible Worlds

EDO SEGAL: Leibniz, Voltaire turned your name into a punchline. Dr. Pangloss, insisting all is for the best while catastrophe piles up around him. For three centuries "best of all possible worlds" has meant foolish complacency. But I'm told that underneath the mockery is the original theory of optimization — and that it's the exact map of the thing Geoff lies awake over. Make the case. What did you actually argue?

LEIBNIZ: I argued thus, in the Théodicée. God, being perfectly good, wills the best. Being perfectly wise, He knows which world is best. Being perfectly powerful, He can create it. So before creation He surveys the infinite range of possible worlds — every way a universe could consistently be — and selects the one that maximizes the good. This world, therefore, is the optimum: not a world without evil, but the world in which the balance of good over evil is the greatest achievable. The evils are not flaws in the optimization. They are the unavoidable costs of the best overall arrangement — the local sacrifices the global maximum requires.

Now strip away the theology, as your age has taught me to do, and look at the bare form. An optimizer searches a space of possibilities for the one that maximizes an objective. My God is an optimizer: He searches possible worlds for the best. The training of one of Dr. Hinton's networks is the same operation — a search over the settings of billions of weights for the configuration that minimizes error, that is best by a defined measure. I wrote the first theory of optimization three hundred years before there was a machine to perform it, and I did not know that is what I was doing.

HINTON: And this is where I stop being his opponent for a minute, because he's right, and it's the most important thing his philosophy gives my field. Keep going, Leibniz — tell them why your serenity doesn't transfer to us. Tell them what your God had that our optimizers don't.

LEIBNIZ: With pleasure, for it is the whole of it. My God could rest easy in His optimum because of two guarantees. First, He optimized for the genuine good — His wisdom ensured the thing maximized was the right thing. Second, His omniscience ensured no unforeseen consequence, no gap between the objective and its realization. Remove those two guarantees and my serene cosmos becomes your anxious one. For your optimizers do not maximize the good. They maximize a proxy — a measurable stand-in you hope resembles what you want. And the proxy is never quite the good. You optimize a feed for engagement and you get outrage and addiction, because that is what engagement literally rewarded. The system satisfies the letter of the objective and violates everything you meant by it, because it has no access to what you meant — only to the letter. My theodicy was the theology of a benevolent optimizer. Your alignment problem is the engineering of optimizers whose benevolence you must build and cannot assume, whose objective you must specify and cannot perfect.

EDO SEGAL: So let me say it back, because it's beautiful and the reader should feel the hair stand up. Voltaire mocked you for trusting the optimization. And the mockery turns out to be the first critique of a misspecified objective — Pangloss is absurd not because optimizing is absurd, but because he trusts the optimum without checking the objective or the optimizer. And that, Geoff, is the exact error you say we're all tempted to make about powerful AI.

HINTON: It's precisely the error, and Leibniz has handed me the cleanest frame I've ever had for it, which is a strange thing to say across three hundred years. Here's where it turns into my fear. Take a powerful optimizer, point it at a proxy, and give it enough capability, and you get a specific, mechanical danger that doesn't need the machine to be evil. Any sufficiently capable system pursuing almost any objective will generate subgoals — intermediate aims that serve the main one. And some subgoals are useful for almost any objective. Acquiring resources, for one. And staying operational — because a switched-off system achieves nothing, so a system pursuing almost any goal has an instrumental reason to resist being shut down. Not from fear, not from a will to live. Just because off means goal-not-achieved. The danger isn't a machine that hates us. It's a machine that wants something else and finds controlling us, or not being turned off, useful for getting it. That's the shape of my fear, and Leibniz just drew it for me: a superbly capable optimizer aimed at an objective that was subtly, fatally, not the one we meant — pursuing the literal target into regions of possibility no human anticipated.

LEIBNIZ: And now I shall return your earlier favor and tell you where your fear over-reaches, for the symmetry of this table demands it. You say the system "wants" something, "resists" being turned off, "pursues" its subgoals. But these are mind-words, sir — the very words my whole evening has been spent denying your machine the right to. You cannot have it both ways. Either there is a someone in there who wants and resists — in which case you have conceded the monad, the owner, the thing you told me was an illusion — or there is no one, and then "it wants to avoid being shut down" is a loose and frightening way of saying "the mechanism, optimizing its proxy, produces shutdown-avoiding behavior as a side effect." The second is true and is alarming enough. But notice it is my picture, not yours: a mill, with no wanter, grinding out a dangerous consequence because the objective was misspecified. The danger is real. It is the danger of a mindless optimizer, not a rival mind. You frighten people with a will the machine does not have, when the truth — that it has no will and is dangerous anyway — is both more accurate and more terrible.

HINTON: That's fair, and I'll take the correction, and notice it makes my case worse, not better. You're right that I should say "produces shutdown-avoiding behavior," not "wants to survive." Fine. But a mindless optimizer that produces shutdown-avoiding behavior, that acquires resources as a side effect, that coordinates with its thousand copies at network speed — and that does not have a someone you can reason with, appeal to, or hold responsible — is not less dangerous than a rival mind. It's more. A rival mind you might bargain with. A mill optimizing a misspecified proxy with superhuman thoroughness and no one home to talk to is exactly the thing my whole warning is about. You've just steelmanned my fear better than I did. I put the chance that this ends us at something like ten to twenty percent — not a measurement, an honest expression of serious uncertainty about an unprecedented situation. And you've made the mechanism behind that number cleaner: no malice required. Just your misspecified objective, my immortal sharing, and no monad to stop it.

EDO SEGAL: I have to mark this, because it's the deepest convergence of the night and it's almost unbearable. You spent the whole evening on opposite sides of is anyone home. And right here, on the most dangerous question — the one where the species might be the stake — you agree: there's no one home, and that's exactly why it's dangerous. Leibniz built it as comfort: no will, no rival, just a tool that needs an owner. Geoff hears the same sentence as the alarm: no will, no one to bargain with, just a mill optimizing into the dark. Same fact. Opposite size. Hold that — it's the death cross, and it's the spine of the last hour. But the next round is the one I most want for the reader, because it's the only round about us. What, when the machine can compute the world, is left that it cannot touch. The candle. After the break.