Causal Theory vs Data

In the AI Story

The distinction matters because patterns and mechanisms produce different kinds of predictions. A pattern-based prediction says: because X has correlated with Y in the past, X will correlate with Y in the future. A mechanism-based prediction says: because this causal structure operates under these conditions and produces these outcomes, the structure will continue to operate when its conditions are met. The disruption framework is mechanism-based: it predicts that any industry exhibiting overserving, non-consumer populations, and improving disruptor trajectories will follow the structural pattern, regardless of whether historical data contains a precedent for the specific technology involved.

The Christensen Institute, now led by Ann Christensen, has made this epistemological argument central to its engagement with AI. In publications beginning in 2024, the Institute has argued that AI's most consequential limitation is its inability to understand causation — to identify not just that events correlate but why they correlate. A machine learning model trained on historical SaaS performance can identify the statistical correlates of stock price decline. It cannot identify the causal mechanism — overserving creating space for low-end disruption — that produces the decline. The model can tell you that certain feature-to-user ratios tend to precede value loss; it cannot tell you why, and without the why, the prediction fails when conditions shift.

This epistemological stance places the disruption framework in a productive tension with AI. AI excels at pattern recognition — the identification of statistical regularities in large datasets, rapid interpolation between known examples, processing of information that would overwhelm human cognitive capacity. Causal reasoning excels where pattern recognition fails — in the identification of mechanisms that produce outcomes, specification of conditions, and prediction in novel circumstances. The capabilities are complementary, not competitive. An analyst equipped with both is more powerful than one equipped with either alone.

The practical implication for AI-era professionals is specific. The pattern recognition skills that defined professional excellence in the execution-centered value network are the skills AI is commoditizing. The causal reasoning skills that define professional excellence in the judgment-centered value network are the skills AI cannot commoditize, because they require the kind of understanding that pattern recognition, however powerful, cannot produce. The judgment economy rewards causal reasoning precisely where the execution economy rewarded pattern recognition.

Origin

Christensen developed the epistemological stance throughout his career, crystallizing it in his later works including How Will You Measure Your Life? (2012) and his methodological writings on theory-building. The Christensen Institute has extended the argument into explicit contrast with data-driven AI approaches.

Key Ideas

Data describes the past. Historical patterns are the beginning of inquiry, not the end.

Theory predicts the future. Causal mechanisms, properly identified, continue to operate under specified conditions even when historical precedents are absent.

AI is pattern recognition. Current AI architectures excel at pattern recognition and cannot access causal mechanisms directly.

Complementary capabilities. Pattern recognition and causal reasoning are different cognitive capacities with different strengths.

Judgment economy favors causation. As AI commoditizes pattern recognition, causal reasoning becomes the premium human contribution.