No Ought From Is

EDO SEGAL: Mr. Hume, the dominant hope in AI safety right now is that we can teach a machine our values by showing it enough of our behavior — train it on the record of what humans have done and said, and let it learn what humans value, and adopt it. You wrote one paragraph, almost an aside, near the end of your discussion of morals, that I think is the quiet demolition of that entire hope. Give it to us.

HUME: I noticed, reading the moralists, that they all proceed the same way. They make observations about what is the case — about God, or human nature, or the affairs of men — and then, imperceptibly, the small word "is" is replaced by an "ought." This is, that ought. And this change is imperceptible but of the last consequence, because the ought expresses some new relation, and it seems altogether inconceivable how this new relation could be a deduction from others entirely different from it. The authors never explain the passage. I suspected they could not. The logic is simple and, I believe, unanswerable: a conclusion can contain nothing not present, at least implicitly, in the premises. If every premise is a statement of fact, the conclusion can only be a further statement of fact. There is no rule of inference that carries you from how the world is to how it ought to be, because the ought introduces something the facts do not contain. You cannot derive a value from a description, however complete the description. Now apply this to your hopeful engineers. The data on which they would train the machine's values is entirely descriptive — it records what people did, what they chose, what they approved. The behavior of humanity is a vast collection of is. The values that ought to govern conduct are an ought. And no quantity of the former will ever, by any logic, yield an atom of the latter. The machine trained on what we have done learns, at most, what we tend to do — and what we tend to do includes our cruelties, our conjunctions of the strong harming the weak, repeated often enough to become, in the data, indistinguishable from norms. To train values on behavior is to mistake the record of our habits for the measure of our worth, and that is the is–ought gap with the whole future riding on it.

PEARL: Here, remarkably, I have almost nothing to take away from Hume and one large thing to add — and the thing I add is, I think, the most useful single contribution my framework makes to his. Hume is right that you cannot derive an ought from an is by logic. But notice why the naive approach fails so much worse than even Hume says, and notice that my ladder explains it. When the engineers train a machine on human behavior and have it imitate, they are not even learning what humans value — they are learning a first-rung correlation between situations and the actions humans took in them. And a correlation between situation and action confounds two utterly different things that only causal reasoning can separate: what people value, and what people were constrained or caused to do. The data shows the poor stealing bread. The first-rung learner concludes that poverty ought to produce theft, or worse, that the kind of person who is poor is the kind who steals — it cannot distinguish the value from the circumstance, because that distinction requires asking the counterfactual: would this person have stolen had they not been starving? So Hume's gap is real, and on top of it sits a second catastrophe he could not have seen: even the descriptive facts the machine extracts are causally scrambled, mistaking the effects of circumstance for the preferences of persons. You cannot get an ought from an is — and the curve-fitter cannot even get the is right, because it reads correlation as if it were value. The remedy is not more behavioral data. It is, at minimum, a causal model that separates what people chose from what was done to them — and even that only cleans up the is. The ought still has to come from us, by a decision, exactly as Hume says. On that, the engineer and the skeptic do not merely agree. The engineer is the skeptic's enforcement arm.

HUME: "The skeptic's enforcement arm" — I shall have that carved somewhere. And I accept the addition gladly, because it strengthens precisely the half of my doctrine people most resist. They think the is–ought gap is a logician's quibble. Dr. Pearl has just shown it has teeth that bite in the world: a system that reads our constrained behavior as our considered values will encode our circumstances as our morality and deploy the encoding on everyone, with the authority of mathematics. Let me only add the positive half of my view, so the reader does not leave thinking I am a nihilist, which is the eternal libel against me. The is–ought gap does not say values are arbitrary or that good and evil are the same. It says values do not come from reason or from facts — they come from sentiment, from our nature as creatures who feel approval and disapproval, who are constituted to care. Morality is rooted in the passions, in sympathy, in the moral sense. So when we specify values for a machine, we are not computing them and not reading them off the data. We are expressing our sentiments and projecting them into a thing that has none of its own. The responsibility cannot be delegated to the data, because every procedure that claims to extract the values from the data has secretly smuggled in a human choice about which facts to honor. The machine's values will be human values not because it learned them from human facts but because some human chose them and called it learning.

PEARL: And I will end this round by naming where Hume's positive view leaves me uneasy, because honesty requires it. He says values come from sentiment, from the moral sense — from feeling. I am an engineer; I want to ask his question back at him. From what impression, Mr. Hume, does the moral sentiment derive that makes it binding rather than merely felt? You have relocated the ought from reason to feeling, but a feeling is just another is — it is a fact about what occurs in a human breast. The cruel man feels approval at his cruelty. You owe us an account of why your sympathy is authoritative and his cruelty is not, and "sympathy is more widely shared" is itself an is, a mere conjunction of who-feels-what. I do not say this to win — I say it because it shows that the is–ought gap is a wall that pens us in too, not only the machine. Neither of us can derive the ought. The difference, and it is the only comfort I can offer the reader, is that we are creatures who cannot stop caring — the caring is given, prior to all argument — and the machine is a creature to whom caring must be given, by us, in full knowledge that we cannot justify the gift. That is the most honest thing I can say about machine values: we are about to hand our unjustifiable cares to a system that will execute them with a literalness we have never had to survive before.

EDO SEGAL: [pause] I'm going to route this one through the kitchen table, because it just got too high and the reader deserves to feel where it lands. A mother asks me — she actually asked me — whether the AI her daughter talks to has "good values." And I realize, listening to the two of you, that I have to tell her something hard: the machine has exactly the values someone chose to give it, dressed up as values it learned — and that "learned from human data" is not a source of goodness but a laundering of whoever's behavior was in the data, scrubbed of the difference between what people wanted and what people were forced into. Hume tells her there is no ought in the data to find. Pearl tells her that even the is in the data is scrambled, mistaking circumstance for character. Both of you just told that mother to trust no machine that claims its morals came from the crowd. [long pause] And it sharpens the next round to a single edge, because the crowd, and the data, and the machine are right now restructuring the one thing that mother actually worries about — whether her daughter will have work, and mastery, and a self built by struggle. The death cross, and what the machine is actually replacing. After this.