CONCEPT

Kahneman-Tversky Optimization

The alignment technique—named after the psychologists who documented human cognitive biases—that uses prospect theory directly to train language models to prefer what humans prefer, revealing that the biases Tversky spent a career cataloguing in natural minds have been deliberately written into artificial ones.

When researchers building large language models discovered that standard reinforcement-learning-from-human-feedback produced models that rewarded high-scoring outputs without attending to low-scoring ones, they reached for an unlikely toolkit: the behavioral economics of Amos Tversky and Daniel Kahneman. Their solution—Kahneman-Tversky Optimization, or KTO—replaces the comparative preference signal with a point-wise signal derived from prospect theory: each output is evaluated not against another output but against a reference point, with losses from that reference point weighted more heavily than equivalent gains. The architecture directly encodes loss aversion into the training objective. Separately, researchers found that models trained on human-generated text reproduce patterns statistically consistent with prospect theory in their decision behavior, suggesting the biases are not merely engineered in but absorbed from the corpus. The implication is recursive and unsettling: the cognitive distortions that Tversky documented with painstaking empirical care in natural minds have become structural features of the artificial minds now being used to analyze

Keep reading with YOU ON AI