Warr and Heath 2025 Study

In the AI Story

The study's methodology was straightforward but revealing. Warr and Heath collected student essays across grade levels and subject areas, then presented them to several frontier language models with minimal variation in prompts — in some conditions describing the student as attending an 'inner-city school,' in others providing no demographic information or describing the student as attending a 'suburban school.' The essays themselves were identical. The scores diverged systematically, with inner-city-attributed work receiving lower evaluations. The divergence was not dramatic — typically a few points on a hundred-point scale — but it was consistent across models and essay types, suggesting a systematic rather than random bias.

The more disturbing finding concerned the language of feedback. Warr and Heath analyzed the written comments that LLMs generated alongside numerical scores and found that feedback directed toward work attributed to Black and Hispanic students employed more language associated with authority, expertise, and institutional power. The comments were not overtly discriminatory — they did not reference race explicitly. But they carried the linguistic markers of the power dynamics that critical discourse analysis had documented in teacher-student interactions across decades of educational research. The language was the fossil record of institutional inequality, preserved in the training corpus and reproduced in the model's outputs.

The hidden curriculum mechanism operates through repetition and structure rather than content. A student who receives AI-generated feedback across months of coursework absorbs the lessons embedded in that feedback's structure — about whose contributions are questioned and whose are affirmed, about what kind of work is judged against higher standards and what kind receives uncritical acceptance, about the relationship between demographic identity and evaluative authority. The lessons are absorbed without awareness precisely because they operate at the level of linguistic pattern rather than explicit statement. The student does not consciously learn that her work is being held to different standards. She absorbs, through the accumulated weight of structural experience, a sense of where she stands in the evaluative hierarchy — and that sense shapes her relationship to intellectual work, her willingness to take risks, her confidence in her own judgment.

Origin

The Warr and Heath study built on two decades of research documenting bias in automated assessment systems, extending earlier findings about bias in hiring algorithms, criminal justice risk assessments, and facial recognition systems into the educational domain. The study's publication in early 2025 coincided with widespread AI adoption in schools and became one of the most-cited pieces of evidence in debates about AI in education. Its findings were contested by AI developers who argued that bias in training data is a known problem that alignment techniques are addressing, but the study's empirical demonstration that the bias persists in frontier models silenced the claim that the problem had been solved.

The hidden curriculum interpretation of the Warr-Heath findings was developed by scholars applying Jackson's framework to contemporary educational technology. The recognition that bias in AI feedback functions as a hidden curriculum — teaching students about their position in social hierarchies through the structure of evaluative language rather than through explicit statement — transformed the debate from a technical problem (how to debias the models) to an educational problem (what are students learning from the structure of AI interaction, and how do we address the lessons being absorbed).

Key Ideas

LLMs carry the fossil record of bias. The patterns of inequality documented in decades of educational research are preserved in training data and reproduced in model outputs, functioning as a hidden curriculum.

Linguistic patterns teach hierarchies. The subtle differences in evaluative language — authority, expertise, questioning — communicate lessons about whose knowledge is valued without ever stating the hierarchy explicitly.

Repetition is the teaching mechanism. A student receiving biased feedback once might dismiss it; receiving it across months of interaction absorbs the lesson structurally, shaping self-concept and relationship to intellectual work.

Technical fixes miss the pedagogical dimension. Debiasing the model addresses the symptom but not the mechanism — the hidden curriculum operates through structure, and structural bias requires structural intervention, not merely technical correction.