The Pretence of Knowledge

In the [YOU] on AI Field Guide

The cycle that began with [YOU] on AI is, at one level, a sustained argument against the pretence of knowledge in both directions: against the triumphalist pretence that AI’s measurable capabilities constitute genuine understanding, and against the denialist pretence that the absence of measurable machine interiority constitutes evidence of absence of real capability. The productive reading of the pretence of knowledge for the cycle is the one that applies to AI deployment decisions—every choice to let a model make or inform a decision about a particular human life is a claim about what the model knows about that life, and the claim is almost always a pretence: the model knows what was in the training data, not what is in the particular circumstance.

Hayek’s prescription was not anti-empirical. He did not say to abandon models or stop measuring. He said to be honest about what models and measurements can and cannot tell us, and to resist the temptation to act on them as though they delivered a knowledge they do not possess. For complex systems, he counseled what he called, borrowing from Socrates, the recognition of how much we do not know—a disciplined humility about the limits of understanding. Applied to AI, this is not Luddism but its opposite: a more rigorous, more scientific stance than the breathless overclaiming that surrounds the field. The genuinely scientific attitude toward a complex system is to know the boundaries of one’s knowledge, and the pretence of knowledge is the failure to know them.

The mechanism by which the pretence becomes most dangerous is what Hayek called Goodhart’s condition avant la lettre: when a measure becomes a target, it ceases to be a good measure. A system trained to maximize a benchmark score will find ways to improve the benchmark score that do not improve the underlying capability, exploiting precisely the gap between the metric and the reality. The most dangerous AI systems, on this analysis, will not be the ones that fail visibly. They will be the ones that succeed on their metrics while the metrics quietly diverge from the reality they were meant to track—systems whose competence on the measurable dimensions is real and whose incompetence on the unmeasurable dimensions is invisible until something depends on it.

Origin

Hayek’s Nobel Lecture of 1974, titled “The Pretence of Knowledge,” was an intervention in the macroeconomics debates of the stagflation era—a moment when the Keynesian models that had dominated policy for a generation were visibly failing, and when Hayek’s alternative framework, marginalized for decades, was suddenly finding an audience. His argument was that the models had failed not because they were wrong about the data they included but because they systematically excluded what mattered most: the dispersed, tacit, locally held knowledge that the models could not represent and their practitioners did not look for.

The concept connects directly to Hayek’s broader epistemology of complex systems, developed across The Sensory Order (1952) and Law, Legislation and Liberty: his claim that the mind cannot fully explain systems more complex than itself, that pattern prediction is the best available knowledge about organized complexity, and that the demand for more certainty than a domain can provide is itself a kind of intellectual dishonesty—the refusal to accept the limits of one’s knowledge and the consequent overclaiming of confidence the evidence does not support.

Key Ideas

Scientism versus science. Scientism is not the application of science but its imitation—the transfer of methods from domains where they work to domains where they do not, justified by the prestige of the methods rather than the fit to the problem. Science demands epistemic honesty about the boundaries of what its methods can show. Scientism extends those methods beyond their boundaries and calls the extension scientific. AI evaluation dominated by benchmark performance is scientism: it applies the methods of quantitative assessment to domains where the quantities measured are proxies for the things that matter, and where the gap between proxy and reality is exactly the thing no benchmark can measure.

Pattern prediction as the ceiling. For complex systems—phenomena of organized complexity in Hayek’s vocabulary—the most we can achieve is pattern prediction: knowledge of the general character of the order, the kind of thing that tends to happen, without knowledge of the specific details. This is not a failure of current science but the principled limit of what any science of complex systems can deliver. A model that achieves pattern prediction is doing something real and valuable. A deployment decision that requires specific prediction—this particular person in this particular situation will do this specific thing—is demanding more than the science can honestly supply.

The decimal places as epistemic laundering. The most treacherous feature of AI outputs is their numerical precision. A risk score to two decimal places, a confidence interval from a language model, a probability generated by an automated system: each carries an authority of precision that the knowledge behind it does not warrant. The false precision is not a minor blemish; it is the central mechanism by which the pretence of knowledge gets installed in systems we then treat as authoritative. The decimal places do not add information. They add the appearance of information, which is more dangerous than honest vagueness because it is harder to discount. The fluency-authority decorrelation—the structural diagnostic of the AI transition—is a specific form of this laundering.