Vanity Metrics

In the AI Story

Build velocity was weakly actionable in the pre-AI regime because it was constrained by team capacity. An increase indicated process improvement. In the AI-assisted regime, build velocity is determined by the AI tool's capability, and the team's contribution to the metric is mostly the decision about which tool to adopt. The metric has become a measure of tool selection rather than work quality.

Speed metrics — time to first prototype, time to ship, cycle time from idea to deployment — are similarly compromised. Speed of building is not speed of learning. A team deploying new prototypes daily is building fast; whether it is learning fast depends entirely on whether deployments are driven by hypotheses and whether results are analyzed with sufficient rigor to generate insight.

Sophistication metrics — complexity of AI-generated code, number of integrations, breadth of technology stack — measure tool-chain capability rather than customer value. A product integrating seven APIs and three machine learning models is not necessarily more valuable than a product using one API and a simple database. The customer does not care about architectural elegance; the customer cares whether the product solves her problem.

The actionable metrics in the AI age are those that capture the team's capacity for judgment, learning, and strategic direction. Hypothesis resolution rate captures the essential activity. Assumption inventory reduction tracks learning against the venture's actual risk surface. The ratio of experiments analyzed to experiments conducted reveals whether learning debt is accumulating. The ratio of strategic pivots to reactive adjustments distinguishes direction from drift. These metrics are harder to track, harder to display on a dashboard, and harder to celebrate at team meetings — but they are the metrics that matter, because they capture the activities AI cannot perform on the team's behalf.

Origin

Ries introduced the actionable-versus-vanity distinction in The Lean Startup (2011), drawing on observations of startups that pointed to rising totals as evidence of progress while failing to articulate what decisions those totals should inform. The distinction became one of the methodology's most widely cited contributions to practical entrepreneurship.

The AI-era expansion of the category has been driven by practitioner observation. Developers at Anthropic and elsewhere have noted internally that velocity metrics, while widely reported, increasingly reflect tool adoption curves rather than team performance — a pattern consistent with Goodhart's Law operating on engineering management.

Key Ideas

Actionable metrics inform decisions. If this number changes, we will do something specific; if it does not, we will not.

Vanity metrics comfort. They produce the feeling of progress without the information required to improve it.

Production velocity has become vanity. In the AI age, build speed measures the tool rather than the team.

New actionable metrics measure learning. Hypothesis resolution, assumption reduction, analyzed-to-conducted ratio, strategic-to-reactive ratio.

Dashboards must be rebuilt. The pre-AI metric set is not invalid but is insufficient; the AI-age dashboard must add learning metrics alongside production metrics.

Debates & Critiques

Growth practitioners often defend velocity metrics as leading indicators — on the theory that speed-of-shipping correlates with speed-of-learning in well-functioning teams. Ries's response is that the correlation was real in the pre-AI regime when shipping required learning, and has broken down in the AI-assisted regime where shipping can occur without learning. The leading indicator has become a lagging indicator of tool adoption.