Predictive Harm Invisibility

Origin

The concept crystallizes from Narayanan's engagement with the Fragile Families Challenge—a large scientific collaboration in which hundreds of researchers were given a rich dataset following thousands of families over years and challenged to predict life outcomes. The best models predicted only slightly better than a simple formula using a handful of variables. But this scientific verdict produces no corrective pressure on the marketplace, because the vendors of predictive AI tools can point to their systems' apparent accuracy on historical data (which the systems have already seen) without any obligation to demonstrate real-world performance improvement. The historical-data accuracy is the demonstration; the deployment is the product; and Narayanan's method is precisely the refusal to confuse the two.

The invisibility is not accidental. It is a structural feature of the business model. A vendor of a hiring algorithm does not disclose that its core task—predicting which candidate will succeed in a role—is close to impossible according to the best available science, because the disclosure is not required and the harm it conceals is not directly attributable. The candidate who was not hired cannot sue for the opportunity she was denied. The defendant who was held longer cannot demonstrate that he would have been law-abiding if released. The patient who received less care cannot observe whether more care would have been beneficial. In each case, the person harmed lacks access to the counterfactual that would make the harm legible as harm.

The concept connects to the self-fulfilling prophecy at the level of mechanism: when a predictive system's forecast is used to make a decision, the prediction changes the outcome, confirming itself regardless of its accuracy. The high-risk defendant is treated differently, and the treatment alters the risk. The loan applicant denied credit is pushed toward the hardship the model predicted. The prediction is not a neutral observation of an independent future; it is an intervention that helps create the future it claims to describe. This reflexivity means that even the feedback that does become available tends to confirm the prediction, not because the prediction was accurate but because the prediction was enacted.

Key Ideas

The counterfactual deficit. Every predictive AI harm involves a counterfactual that the prediction has foreclosed. The person filtered from a job, scored for risk, denied coverage, or flagged for surveillance cannot access the world in which the prediction was not made and observe whether that world was better. This makes the harm structurally different from harms produced by generative AI (which are visible in the output) and requires a different accountability regime: rather than waiting for harm to be demonstrated, regulatory frameworks must require demonstrating that the system improves real decisions before deployment.

Broken institutions as the vector. Predictive harm invisibility explains why broken institutions adopt flawed predictive systems. The opacity that protects the vendor's product from scrutiny also protects the institution from accountability: when the algorithm rejects the candidate, denies the coverage, or extends the sentence, the decision appears to have been made by an objective technical process rather than by a human agent who could be held responsible. The flawed automation does not merely hide the harm; it transfers apparent responsibility from humans who can be questioned to algorithms that cannot. The invisibility is a service the institution is paying for.

Regulatory implications. The standard instrument for consumer protection—requiring demonstrated harm before intervention—is inadequate for predictive AI harms because the demonstration requires observing the counterfactual. Narayanan argues that the appropriate response is to require demonstration of benefit before deployment: vendors must show that the system improves real decisions about real people, not that it performs well on historical benchmarks, before it can be used in consequential settings. This reverses the burden of proof in a way that matches the structure of the harm.

Arvind Narayanan