The Is-Ought Gap

In the [YOU] on AI Field Guide

The [YOU] on AI cycle asks what it means to live well among machines. The is-ought gap forces the most fundamental answer: living well requires making normative choices that cannot be outsourced to data, derived from behavior, or delegated to a system that will make them automatically. Every value a machine acts on is there because a human put it there, crossing the gap by an act of will rather than a derivation. The pretense that values are learned from data disguises a profound responsibility as a technical procedure.

The gap also explains a structural feature of the AI moment that the cycle returns to repeatedly: the tendency to treat descriptive AI capabilities as if they established normative conclusions. The observation that a system can perform a task better than a human does not entail that the system ought to perform that task, or that the human ought to defer to it, or that the arrangement is good. These conclusions require normative premises that are not present in the performance data. Hume’s point is that the inference is invalid as stated, and that making it invisible does not make it valid.

The cycle’s practical upshot is that value specification is irreducibly political, ethical, and human. Who decides what the machine ought to optimize? What process legitimates that decision? Whose values are represented in the training data treated as authoritative? These are not engineering questions awaiting technical solutions. They are normative questions that must be answered normatively, by human beings who accept responsibility for crossing the gap—and who cannot, by invoking the sophistication of the technical apparatus, pretend the crossing did not happen.

Origin

The observation appears in section 3.1.1 of Hume’s Treatise of Human Nature (1739), in the opening book of his discussion of morals. Hume had spent the previous sections arguing that moral distinctions are not derived from reason but from moral sentiment—from the feelings of approval and disapproval that arise from our constitution as social creatures. The is-ought observation is the negative side of that doctrine: since values come from neither reason alone nor the facts of the world, they must arise from the passional nature that is prior to any calculation.

The observation was not widely noticed in Hume’s own century; it became central to twentieth-century metaethics under the label “Hume’s guillotine” or the “naturalistic fallacy” (the latter term coined by G.E. Moore in 1903 for the related error of equating ‘good’ with any natural property). In the twenty-first century, as AI researchers attempt to specify machine values from training data, the gap has acquired new urgency and new engineering stakes. It is no longer a puzzle for philosophers alone but a structural obstacle in the most consequential technical project of the age.

Key Ideas

The logical structure. Valid deductive arguments cannot produce conclusions that go beyond what is present in the premises. Factual premises yield only factual conclusions. Normative conclusions require at least one normative premise. Any argument that moves from a purely factual description to a normative conclusion has made an implicit normative assumption that should be made explicit and examined. When that assumption is hidden—as it typically is in discussions of what AI “should” do based on data about what people have done—the responsibility for the assumption is also hidden.

Applied to preference learning. The dominant approach to AI value alignment involves learning from human preferences—ratings, rankings, demonstrations, feedback. Hume’s gap explains why this approach cannot, by itself, close the value specification problem. The data encodes what people have preferred; the system learns to predict and replicate those preferences; the claim that those preferences are what ought to be pursued is a normative addition supplied by the designers’ decision to treat observed preferences as authoritative. That decision is itself a value choice—the choice to defer to revealed preference rather than considered preference, to the preferences of those in the training set rather than those excluded from it. The is-ought gap is not bridged by more data; it is crossed by the decisions about which data counts.

Responsibility follows. If values cannot be derived from facts, every value a machine acts on was put there by a human choice. This means the responsibility for those values is fully human and cannot be dissolved into the sophistication of the technical process. An organization that deploys a system with values that cause harm cannot claim the system “learned its values from data;” it chose which data to treat as authoritative, chose how to weight conflicting preferences, and chose when the system was value-aligned enough to deploy. The is-ought gap makes these choices visible as choices rather than as derivations, and assigns responsibility accordingly.