CONCEPT

The Coded Gaze

Joy Buolamwini’s term for the partiality embedded in machine-perception systems—not the product of malicious intent but of homogeneous builders who treated their own partial sample of humanity as the universal default.

The coded gaze is a theory of how bias enters technical systems, and it borrows deliberately from a longer intellectual tradition about who gets to look and who is looked at. To speak of a gaze is to insist that perception carries a perspective—that the act of seeing is also an act of judging, and that systems built to perceive the world will encode the judgments of their makers whether or not anyone intended them to. The concept’s force comes from its insistence that the bias is structural rather than incidental: the coded gaze does not require a bigoted author, only a homogeneous one. A team that draws on its own experience to decide what to build and how to test it unconsciously treats its own kind as the default human, and the system that results works beautifully for people like its makers and poorly for everyone else—without anyone in the room having meant any harm. This reframing matters because it changes what a solution looks like: not removing bad individuals but widening the circle of who builds, whose data is collected, whose use cases are imagined, and whose failures are noticed—because the gaze will never widen on its own. Buolamwini arrived at the concept through a costume mask she held over her face to make a facial-analysis system recognize her, an image she has called both literally true and unavoidably symbolic: the partial view had been deployed as whole vision, and it took someone standing outside the gaze to make it visible at all. Applied to today’s generative systems, the coded gaze predicts much of what we observe: image models that reach for certain faces when asked to depict a doctor or a criminal, language models that treat some dialects as standard and others as deviation—the statistical sediment of a world unevenly recorded, deployed as if it were the world entire.

In the [YOU] on AI Field Guide

The cycle’s governing metaphor describes AI as an amplifier that carries whatever signal it receives further and faster than any previous tool. The coded gaze names what is in the signal before the amplifier picks it up: the accumulated partiality of who built the system, what data they gathered, and which failures they were positioned to notice. An amplifier carrying a biased signal does not merely transmit bias at the original level—it transmits it at scale, with the authority of computation, and without the face of any human prejudice to confront or argue with.

The concept is also a precise account of why the fluency-authority decorrelation falls unequally. The same confident fluency that the cycle identifies as the signature hazard of the age is distributed differently across populations: the system is most fluent, most accurate, most authoritative for the people it was implicitly built for and least reliable—yet equally confident—for the people who fall outside the gaze. The miscalibration is invisible from inside.

Origin

Buolamwini coined the phrase while writing about the white-mask incident at the MIT Media Lab and the Gender Shades research it prompted. She drew deliberately on feminist and postcolonial traditions of gaze theory—particularly the insight that looking is never innocent, that every apparatus of vision encodes the position of the one who built or trained it. Her contribution was to show that this insight applies with full force to algorithmic systems: they do not transcend perspective by virtue of being mathematical; they fix perspective in place and then amplify it.

The Gender Shades study provided the empirical demonstration. Commercial facial-analysis systems from IBM, Microsoft, and Face++ achieved error rates as low as 0.8 percent for lighter-skinned men and as high as 34.7 percent for darker-skinned women. The disparity was not a rounding error—it was a chasm, and it fell precisely along the lines that the coded gaze predicts: greatest failure at the intersection of race and sex, for the group least represented in the training data and least present in the teams that built and tested the systems.

Key Ideas

Structural, not incidental. The coded gaze reframes algorithmic discrimination from an individual failing to a systemic one. It asks not “who typed a prejudiced rule?” but “who was in the room, whose data was collected, whose use cases were imagined?” The answer to those questions determines the gaze; and homogeneous answers produce a narrow gaze that cannot see its own limits.

Self-perpetuating. The coded gaze reproduces across generations of technology because new systems are trained on data shaped by old assumptions, evaluated by benchmarks built under those assumptions, and refined by teams selected by institutions that reproduce themselves. Left unexamined, the partiality hardens into infrastructure: no longer anyone’s assumption but the neutral default of the system, present in every output.

Generalizes beyond faces. Buolamwini introduced the concept through facial analysis but was always clear that it names a general mechanism. Every large model trained on scraped text and images is a vast sample of the world dressed up as knowledge of the world. It reflects the languages most represented online, the perspectives most often written down, the aesthetic and political assumptions of the corpora it consumed. The coded gaze is what you get when a partial sample is deployed as universal vision—and that description applies to every contemporary AI system.

Debates & Critiques

The central debate about the coded gaze is whether it describes a solvable problem or an inherent feature of any system trained on human-generated data. Optimists argue that diversity in teams, balanced benchmarks, and careful data curation can substantially close the gap—and point to the corrections that followed the Gender Shades study as evidence. Skeptics, including Buolamwini herself, note that successive generations of systems have been built with greater stated awareness of the problem and have still exhibited it, suggesting the mechanism runs deeper than any individual corrective. A second debate concerns scope: some argue that the coded gaze is a technical problem amenable to engineering solutions, while Buolamwini insists it is also a political problem requiring institutional accountability, legal remedies, and the participation of the excoded in defining what counts as harm. Kate Crawford’s political-economy analysis converges on this point: the gaze is not merely a technical artifact but a product of the social relations of AI’s production, and those relations must change for the gaze to widen.

In the [YOU] on AI Field Guide

Origin

Key Ideas

Debates & Critiques

Related Entries

Further Reading