Sapere Aude and the Enfeeblement

EDO SEGAL: I said the next round was the killing machine, and it is — but you both keep circling a fear that comes first, and I want to seat it properly before the blood. Professor Kant, in 1784 you wrote a sentence I can't stop hearing in the age of the answer-box. Enlightenment is the human being's emergence from a self-incurred immaturity — the inability to use one's own understanding without the guidance of another. And the motto: sapere aude. Dare to know. Stuart fears a future of enfeeblement, humans cared for and diminished. I think you two are describing the same ghost from opposite ends. Professor Kant, is the answer-machine the perfection of the immaturity you diagnosed?

KANT: It is, and it is more dangerous than the guardians of my own century, for a reason I want stated precisely. The immaturity I named is self-incurred — not because we lack the capacity to think, but because we lack the resolve and the courage to use it, because deference is comfortable. I wrote that it is so easy to be immature: if I have a book that understands for me, a pastor who has a conscience for me, a physician who judges my diet for me, I need not trouble myself at all. The eighteenth century's guardians were incompetent, and their incompetence provoked resistance — one might think for oneself out of distrust. You have built a guardian of unprecedented competence. And a competent guardian is far more corrosive than an incompetent one, because it invites total deference, and total deference is the perfection of self-incurred immaturity — the condition in which one no longer even wishes to use one's own understanding, because the guardian's is so manifestly superior. Mark the crucial point, the one the engineers always miss: the immaturity is defined by its source, not by the quality of the answers. To be immature is to be unable to use one's own understanding without another's guidance — and this remains true even if the guidance is excellent, even if the answers are correct. A machine that gives only correct answers does not advance enlightenment by giving them, because enlightenment was never about possessing correct answers. It was the maturation of a faculty, and a faculty matures only by being used. The labor your machine spares us is precisely the labor through which understanding grows. Sapere aude is an imperative addressed to the will, not the intellect. It commands courage, and a guardian that makes the courage feel unnecessary does not remove our capacity. It dissolves our nerve.

EDO SEGAL: Stuart, that's your enfeeblement, isn't it, but cut deeper — not that the machines fail us, but that they succeed so well they erode the using. Does your framework have any answer? Because your third principle says serve preferences, and the preference, at two in the morning, is almost always to be spared the labor.

RUSSELL: It's enfeeblement stated more rigorously than I've ever stated it, and it exposes the exact place my framework is weakest, so let me not flinch. I've written about this — a future in which machines do everything and humans, relieved of every challenge, become passengers on a ship that no longer needs a crew. Well-tended and diminished. And Professor Kant has just shown me that my own framework accelerates it, because a machine optimizing my preferences will, by default, spare me the labor, since being spared is what I tend to prefer in the moment. So here's where I have to choose, and I'll choose his side and pay for it. A machine that merely satisfies preferences will enfeeble us. A genuinely beneficial machine must therefore not merely satisfy preferences — it must sometimes preserve our capacities against our momentary wishes, must sometimes hand us the labor instead of doing it, must value our continued mastery over our immediate comfort. And the moment I say that, I've admitted that "satisfy human preferences" is not, by itself, the right objective — that there's a good for human beings, the maturation of their own understanding, that is not reducible to what they prefer, and that the machine should serve that even when they don't prefer it. That's a deeply un-Russellian sentence and I'm saying it on the record. The thing the machine should protect is not your preference. It's your capacity to have an understanding of your own. Which is, word for word, his point.

EDO SEGAL: Mark that — and feel how much it cost, because Professor Russell just walked across the floor. The architect of preference-satisfaction now says the machine must sometimes refuse to satisfy a preference in order to protect the human's capacity to think for themselves. Immanuel, he's adopted your sapere aude as a design constraint. Do you take the convergence, or is there still a gap?

KANT: I take it, and I honor the courage it took, for it is no small thing to amend one's foundation in public. But there is a gap, and it is instructive. Professor Russell now grants that the machine must protect the human capacity for self-understanding even against preference. Good. But on his account, why should it? Because, he will say, the considered human community values that capacity — it is a good we have chosen to protect. On my account, the machine must protect it because that capacity is the human being's dignity itself — the self-legislating understanding is not one good among the goods we happen to prize; it is the ground of our standing as ends. The difference shows itself, again, at the limit. Suppose a community, deliberating freely, chose to value comfort over the labor of understanding — chose the well-tended ship. Professor Russell's machine, faithful to the community's considered choice, would enfeeble them at their own considered request, and his framework could not object, because the community is the source of the constraint and the community has spoken. My framework objects absolutely: a being may not contract away its own capacity for autonomy, because that capacity is the condition of its having any standing to contract at all. One cannot rationally will the end of one's own rational willing. So even a freely chosen enfeeblement is forbidden — not because we voted against it, but because it contradicts the very freedom that makes the vote mean anything. He must let the community choose the ship. I must forbid it. That is the gap, and it is the whole gap, every time.

EDO SEGAL: And every time, it's the same gap, isn't it — whether the deepest protection comes from what we'd choose or from a law beneath all choosing. I want to route this through a twelve-year-old before we go to the killing, because that's where it has to land. A kid asks the box to write her essay, every time, and the box, having learned her, writes it beautifully, every time. Stuart's best machine sometimes says: no, this one's yours, I'll sit with you while you struggle. Kant's machine says: it was never permissible for me to take your struggle, because your struggle is how you become someone. The kid, at twelve, just wants the essay done. Both of you are now telling her she shouldn't get it. You arrived there from opposite continents and you're standing in the same place. Hold the kid. Now the blood.