Explication and the Alignment Problem

Rudolf Carnap called it explication: the operation of taking a concept that is familiar but vague—a concept we use confidently in ordinary life but cannot define, that blurs at the edges and shifts with context—and replacing it with a concept that is precise, exact, and fit for systematic use. The everyday concept he called the explicandum; the precise replacement, the explicatum. To explicate “warm” is to replace the felt, situational notion with a number on a thermometer. To explicate “proof” is to replace the intuitive sense of a convincing argument with the formal, checkable structure of logical derivation. Explication is, Carnap believed, the central task of scientific philosophy. It is also—though he never knew the word—the central task of artificial intelligence. Every time a vague human notion is turned into a metric a system can optimize, Carnap's operation has been performed, whether the person performing it has heard his name or not. The danger