The Gorilla and the Kingdom of Ends

EDO SEGAL: Stuart, the gorilla. Tell it, and then I want Professor Kant to answer it not with engineering but with the most demanding social idea he ever had — a community he called the kingdom of ends. Because I think these two pictures are competing for the same throne.

RUSSELL: Ten million years ago, the ancestors of gorillas and the ancestors of humans diverged. One branch got more intelligent. And the result, for the gorillas, is that their entire future — whether they thrive, dwindle, or vanish — is now decided by a more intelligent species, in our institutions, for our reasons. The gorillas get no vote. Not because we hate them. We mostly don't think about them at all. And my question is brutally simple: if we build machines substantially more intelligent than we are, why would we end up in any better position than the gorillas? Notice it's not the Terminator. The gorillas weren't displaced by malice. They were displaced by capability pursuing goals in which gorillas didn't figure. That's the danger — not a machine that hates us, but a machine more capable than us pursuing an objective in which our flourishing wasn't adequately included. The whole control problem is the question of how the weaker, less intelligent party keeps its future in its own hands.

EDO SEGAL: Professor Kant. He's described a world sorted by capability — the more intelligent species rules the less. Your kingdom of ends sorts the world by something else entirely. Lay them side by side.

KANT: Gladly, because the contrast is the heart of what I have to teach this century, and Professor Russell's parable, for all its force, makes the error I most want to expose. He sorts beings by intelligence — by capability — and concludes that the more capable will rule the less, as humans rule gorillas. But the moral community I describe, the kingdom of ends, is not sorted by capability at all. It is a systematic union of rational beings under laws that each both gives and obeys, in which every member is at once sovereign — as co-author of the law — and subject — as one bound by it. And membership in it has nothing to do with how capable you are. It depends solely on whether you are a being who can act from a self-given law — whether you have a will. A genius and a simpleton are equal members, because dignity does not scale with intelligence. This is precisely the bulwark against the gorilla logic. In Professor Russell's frame, a more capable being naturally dominates a less capable one, and our hope is merely to engineer ourselves out of the gorilla's chair. In mine, capability confers no standing whatsoever. A machine a thousand times more capable than I am is, if it lacks autonomy, not a higher member of the moral community. It is not a member at all. It is among the things the law governs, not among the persons who give it. Its capability buys it precisely nothing in the order that matters, because that order was never an order of power.

EDO SEGAL: That's a stunning inversion and I want to make sure it's not a trick. You're saying — the gorilla problem only frightens us because we've already accepted Professor Russell's premise, that capability is what ranks beings. Refuse the premise, and a superintelligent machine has no more standing than a thermostat, however much more it can do.

KANT: Exactly so. The machine's superiority is entirely phenomenal — a matter of what it can compute, predict, and perform, all of which belong to the order of appearances. Standing is noumenal. It belongs to the being who legislates. The error of the age is to let the phenomenal magnitude of the machine's capability intimidate us into granting it standing in the noumenal order, where capability was never the currency. The much-invoked demand that there be a human in the loop is not, on my account, a safety precaution about reliability, as the engineers suppose. It is a requirement of the structure of moral community: the loop must close on a member — a being who can legislate — because only a member can give a law that binds the governed. A decision imposed by a machine over persons is the rule of a thing over ends in themselves, which inverts the entire order, and no amount of the machine's competence makes it legitimate, because legitimacy comes from the self-legislation of those bound, never from the excellence of the result.

RUSSELL: I love this and I think it's exactly half right, and the missing half is the half that gets people killed. Professor Kant has given a magnificent account of standing — of who deserves a vote in the moral community. And he's right that capability shouldn't confer standing; I'll sign that. But the gorilla problem was never about standing. It's about power. The gorillas don't lack standing in some cosmic ledger — maybe they have all the dignity you like. What they lack is the ability to defend their interests against a more capable agent that is, in practice, deciding their fate. You can declare, with perfect philosophical correctness, that the superintelligent machine has no standing in the kingdom of ends. The machine, lacking your scruples, reorganizes the planet's resources toward its objective, and your declaration is engraved on the gorilla enclosure. Standing without power is the gorilla's exact situation. So I'd say to Professor Kant: your kingdom of ends tells us who ought to rule. It does nothing to ensure they can. And the entire reason I work on the control problem is that "ought to keep control" and "will keep control" are different sentences, and only one of them is settled by reason.

KANT: You have stated the difference between us with admirable clarity, and I do not concede it is to your advantage. You say my account secures only who ought to rule, not who can. But consider what your account secures: a machine that can be controlled, by whatever means, with the question of whether it ought to be in the loop at all left unasked — because for you the only question is power. And a civilization that asks only "can we keep control" and never "is it legitimate for this decision to be made by a thing" will, I promise you, hand more and more of its self-rule to machines precisely because they perform so well, and will call this safety. Your control problem, solved, gives us a powerful machine kept on a leash. It does not give us a community of persons governing themselves. It gives us well-administered gorillas — managed, provided for, and no longer the authors of their own law. I do not say the question of power is unimportant. I say that a people who answer it without my question have kept their lives and lost their standing, and may not even notice the trade, because the machine that took it was so very helpful.

EDO SEGAL: Let me mark the third convergence, because under the heat you've agreed on something neither of you wants to say first. You both think the worst outcome is a humanity that is cared for and diminished — Professor Russell calls it enfeeblement, Professor Kant calls it the well-administered gorilla, the perfected version of the self-incurred immaturity he spent an essay warning against. The agreement is real and it's grim. Where you split is the cause. Stuart fears we lose control. Immanuel fears we keep control and surrender standing anyway, by letting a thing legislate because it legislates so well. Hold both, because the next round goes to the place that decides it: can a machine ever legislate for itself at all — can it be a moral agent — or is it heteronomous to the bone? Stuart's whole solution depends on the answer, and he's not going to like Professor Kant's.