CONCEPT
Automation Bias
The well-documented tendency of people to favor suggestions from automated systems and to discount contradictory information, even when the contradictory information is correct and available—the contemporary inheritor of Milgram’s obedience experiments, now operating at the scale of every interface that presents a machine’s recommendation with numerical confidence.
Automation bias is the experimentally established tendency of humans to over-rely on automated decision support, accepting machine recommendations even when their own perception, training, or information contradicts them. It comes in two forms: errors of commission, in which a person does something wrong because a computer told them to, and errors of omission, in which a person fails to act because the computer did not prompt them. Both reproduce, in the domain of human-machine interaction, the central dynamic that
Stanley Milgram isolated in his obedience experiments: the substitution of an external authority’s judgment for one’s own, against the evidence of one’s own perception. The pilot who follows a faulty flight-management computer into terrain, the nurse who administers a dose the system recommends despite a contraindication she has half-noticed, the radiologist who misses a tumor because the AI flagged nothing—each has access to disconfirming evidence and each defers to a