Lorraine Daston

Foreword

By Edo Segal

The sentence that made me put my laptop down was about a flower.

Not a real flower. A drawing of one. An eighteenth-century botanical illustration, commissioned by a scientist who told the artist: Don't draw the plant on the table. Draw the species. Smooth out the insect damage. Straighten the crooked stem. Remove everything that makes this particular specimen particular, and give me the ideal.

The artist did. The image was beautiful. And for two hundred years, the scientific community treated that beauty as proof that the image was true.

I read that and felt the blood leave my face. Because I had been doing the same thing with Claude for months.

Not with flowers. With ideas. I would describe a half-formed thought, and Claude would return it polished, structured, confident — smoothed of all the awkward particulars that made the thought genuinely mine. And I would look at the beautiful output and feel certain it was right. Not because I had checked. Because it looked like it should be right. Because the surface was flawless.

Lorraine Daston is a historian of science who spent four decades at the Max Planck Institute in Berlin studying a question most of us never think to ask: How did we learn to trust what we think we know? Her answer is unsettling. Every era builds knowledge-producing technologies — illustrations, photographs, statistics, algorithms — and every era mistakes the surface features of those technologies for proof of their reliability. The beautiful drawing looks true. The sharp photograph looks objective. The precise decimal place looks rigorous. Each surface feature activates a learned trust reflex so deep it operates below consciousness.

And then AI arrived, producing prose so fluent it activates every trust reflex that literate culture has spent centuries installing.

This is not a book about technology. It is a book about the oldest trap in the history of human knowledge: confusing how something looks with how much it knows. Daston traced that trap across five centuries and found the same architecture every time — overconfidence, costly errors, and then, slowly, the institutions that teach us to see past the surface.

I needed her framework because my own book, The Orange Pill, diagnosed the seduction but could not fully explain its mechanism. Daston can. She shows that what I experienced with Claude's Deleuze error was not a glitch. It was the latest iteration of a pattern as old as the scientific revolution itself.

The machines have learned to look like they understand. Daston teaches you how to tell the difference.

— Edo Segal ^ Opus 4.6

About Lorraine Daston

1951–

Lorraine Daston (1951–) is an American historian of science whose work has fundamentally reshaped understanding of how scientific knowledge is produced, validated, and trusted. Born in East Lansing, Michigan, she studied history at Harvard, where she earned her PhD, before joining the faculty at Princeton, the University of Göttingen, and the University of Chicago. From 1995 to 2019, she served as director of the Max Planck Institute for the History of Science in Berlin, building it into one of the world's preeminent centers for the historical study of scientific practice. Her landmark book Objectivity (2007), co-authored with Peter Galison, traced the emergence of scientific objectivity as a historically specific concept rather than a timeless ideal, identifying successive regimes of epistemic virtue — "truth-to-nature," "mechanical objectivity," and "trained judgment" — each with its own practices, confidence artifacts, and characteristic blindnesses. Her other major works include Classical Probability in the Enlightenment (1988), Wonders and the Order of Nature, 1150–1750 (2001, with Katharine Park), Against Nature (2019), and Rules: A Short History of What We Live By (2022). A recipient of the Pfizer Prize, the Sarton Medal, and the Balzan Prize for History of Science, Daston is recognized as one of the most influential historians of science of her generation, whose concepts — the moral economy of science, the genealogy of data, the epistemology of trust — have become foundational across the humanities and social sciences.

Chapter 1: The History of Objectivity Is a History of Overconfidence

Before the 1830s, the finest scientific atlases in Europe were produced by a method that would strike any contemporary reader as a form of systematic dishonesty. The botanist who commissioned an illustration of a particular plant species did not ask the illustrator to draw the specimen sitting on the table in front of her — that specific plant, with its insect-damaged leaves, its asymmetrical stem, the particular droop inflicted by last week's frost. She asked the illustrator to draw the species. The ideal form. The plant as it was supposed to look, purified of the contingencies that made any individual specimen a flawed representative of its kind. The illustrator smoothed the irregularities, straightened the asymmetries, corrected the accidents of growth and decay. She produced an image that was, in a sense her era would have recognized immediately, more true than any particular truth.

This was not considered dishonest. It was considered the highest form of scientific representation. The practice had a name — Lorraine Daston and Peter Galison, in their landmark study Objectivity, called it "truth-to-nature" — and it rested on a coherent epistemological foundation. Nature, in its individual instances, was noisy. Every specimen was marred by accidents — damage, disease, the vagaries of soil and weather and the season in which it happened to be collected. The task of the scientific illustrator was to see past these accidents to the underlying form, the essential structure that persisted across individuals, across seasons, across the contingencies of particular lives. The illustrator's trained eye was the instrument. Her skilled hand was the mechanism of selection. The image she produced was not a record of what she saw. It was a synthesis of what she knew — a composite drawn from years of observation, informed by taxonomic theory, disciplined by the conventions of her craft.

The confidence that accompanied this practice was enormous and, within its own framework, entirely warranted. The scientific illustration was trusted because it bore the marks of expertise. The clean lines, the precise rendering of venation and surface texture, the careful attention to three-dimensional form — these features communicated that the person who produced this image understood the subject at a level that casual observation could not reach. The illustration's authority was the illustrator's authority. You trusted the image because you trusted the judgment of the trained expert who made it.

Two things deserve notice about this implicit claim. The first is that it was substantially correct. The trained illustrator genuinely did see things the untrained eye missed. Her botanical knowledge was real. The taxonomic categories that organized her perception were productive — they enabled classification, comparison, the accumulation of systematic knowledge across continents and centuries. The illustrations worked. Species were identified. Floras were constructed. The knowledge accumulated, and much of it still holds.

The second thing to notice is that the claim exceeded the reliability it warranted. The illustrator's judgment was not a transparent window onto the essential form. It was a human judgment, shaped by the aesthetic conventions of her era, the theoretical commitments of her discipline, the institutional pressures of the patron who commissioned the atlas. Different illustrators, working from the same specimen, produced subtly but systematically different images, because their judgments about what counted as essential and what counted as accidental were informed by different training, different theoretical orientations, different schools of botanical thought. The variations were not random. They were patterned. But the images themselves bore no marks of these variations. Each illustration presented itself as the species — a direct representation of the essential form, unmarked by the particular constellation of judgments that had produced it.

The confidence artifact, to coin a term that will recur throughout this book, was the illustration's aesthetic authority. The beauty of the image functioned as a warrant for its accuracy. The more skilled the rendering, the more trustworthy it appeared — and the correlation between skill and accuracy was real enough to sustain the practice for centuries, but imperfect enough to produce systematic distortions that the practice itself could not detect, because the criteria for detecting them were internal to the very tradition that produced them.

This is the structure that repeats. Not the specific content — the technology changes, the institutional settings change, the vocabularies of justification change with each era and each discipline. What persists is the architecture of the cycle: a knowledge-producing technology arrives; its proponents identify it as the method that finally removes the distorting influence of human subjectivity; a period of confident deployment follows, during which the technology's outputs are treated as transparent windows onto reality; and then, gradually, over decades or generations, the technology's own forms of distortion become legible to practitioners who have accumulated enough experience to distinguish between what it reveals and what it conceals.

Daston's research, conducted over four decades at the Max Planck Institute for the History of Science in Berlin, has documented this cycle with a precision and range that make the pattern unmistakable. Her work reveals that "objectivity" is not a timeless ideal that scientists have always pursued. It is a historically specific concept, invented in the mid-nineteenth century, with a particular set of institutional conditions, a particular set of epistemic anxieties, and a particular set of practices designed to address those anxieties. Before the concept existed, scientists pursued other epistemic virtues — truth-to-nature being one of them — and those virtues had their own standards, their own confidence artifacts, their own patterns of warranted and unwarranted trust. The history of objectivity is not a story of progress from subjectivity to objectivity, from bias to neutrality, from error to truth. It is a story of successive regimes of epistemic virtue, each one responding to the perceived failures of its predecessor, each one introducing its own forms of distortion while correcting those of the regime it replaced.

The relevance of this history to the present moment should be apparent, but the specific mechanism by which it applies requires careful articulation, because the parallel is both closer and more complicated than the simple formulation "AI is the latest knowledge technology to be over-trusted" might suggest.

Edo Segal's The Orange Pill describes the experience of working with Claude, Anthropic's AI system, in terms that a historian of objectivity recognizes immediately. He recounts a passage in which Claude connected Csikszentmihalyi's flow state to a concept it attributed to Gilles Deleuze — "something about 'smooth space' as the terrain of creative freedom." The passage was elegant. It connected two threads of the book's argument beautifully. Segal read it twice, liked it, and moved on. The next morning, something nagged. He checked. The philosophical reference was wrong — not approximately wrong, not a matter of interpretation, but wrong in a way that would have been obvious to anyone who had actually read Deleuze.

The episode is diagnostic. The passage worked rhetorically. It sounded like insight. It had the aesthetic authority of polished, well-structured argument — the same aesthetic authority that a truth-to-nature illustration derived from its clean lines and precise rendering. And the aesthetic authority functioned in exactly the same way: it inspired confidence in the content by virtue of the quality of the presentation, and the confidence exceeded the epistemic warrant. The prose was beautiful. The reference was false. The beauty concealed the falsehood, not through any intent to deceive, but through the operation of a deeply learned evaluative heuristic: polished expression signals genuine understanding.

This heuristic has been reliable for centuries. A person who writes with the fluency of an expert on Deleuze has almost always spent years reading Deleuze. The correlation between prose quality and domain knowledge is so strong, so deeply confirmed by experience, that it operates below the threshold of conscious evaluation. Readers do not decide to trust polished prose. They simply do, in the same way that viewers of scientific illustrations did not decide to trust aesthetically accomplished images. The trust was an automatic response to a surface feature that had been reliably correlated with the depth property it was taken to indicate.

What Daston's framework reveals — and what the Deleuze episode illustrates with uncomfortable precision — is that AI has broken this correlation. It produces prose that bears all the surface marks of expertise without the underlying process that makes those marks reliable indicators. The decorrelation is invisible in the output. There is no seam, no telltale mark, no stylistic signature that distinguishes the passage where AI's prose quality reflects genuine accuracy from the passage where it does not. The confidence artifact operates identically in both cases.

The history of truth-to-nature illustration ended not in catastrophe but in calibration. Over decades, practitioners developed methods for evaluating illustrations against specimens, for comparing illustrations produced by different artists, for identifying the systematic biases that particular schools and traditions introduced. Standards emerged. Institutional mechanisms were constructed — comparative atlases, standardized conventions for indicating inferred versus observed features, protocols for independent verification. The illustration did not become perfectly objective. No technology does. But the gap between the implicit claim and the actual reliability narrowed, because the community of users developed evaluative competencies adequate to the technology's specific forms of distortion.

The question that governs this book is whether an analogous calibration can be achieved for artificial intelligence — and the historical evidence suggests both that it can and that it will take considerably longer than anyone currently expects. The mechanisms of calibration are neither automatic nor inevitable. They must be built, institution by institution, standard by standard, competency by competency. And they must be built by people who understand both the technology's genuine power and its characteristic modes of failure — who have studied the river, to borrow a metaphor from The Orange Pill, carefully enough to know where it runs dangerous and where it runs generative.

Daston herself, in a 2022 interview, offered a formulation that cuts to the heart of the matter: "Never in the history of human rulemaking have we created a rule or a law that did not stub its toe against unanticipated particulars." The observation applies with full force to the rules encoded in AI training data, in model architectures, in the reward functions that shape outputs. These are rules — complex, statistical, opaque, but rules nonetheless — and they will stub their toes against unanticipated particulars with the regularity that the entire history of human rulemaking predicts. The question is not whether the stumbling will occur. It is whether the institutions charged with detecting and correcting the stumbles will be adequate to the task — and whether they will be built in time.

The history of objectivity is a history of overconfidence. It is also a history of eventual calibration. The first without the second is a counsel of despair. The second without the first is a fantasy of painless progress. The truth includes both: the overconfidence is structural, built into the relationship between any new technology and its users, and the calibration is achievable, but only through sustained institutional effort of a kind that has never, in the historical record, kept pace with the technology it was calibrating.

The machines have entered the current. The current is changing. And the evaluative institutions that should be directing the flow are, at best, in their earliest stages of construction.

---

Chapter 2: When the Photograph Replaced the Drawing

In the 1840s, a new technology arrived in the laboratories and field stations of European science, and it arrived with a promise so compelling that it reorganized the epistemological landscape within a generation. The photograph offered something no previous representational technology had offered: the apparent removal of the human from the process of recording nature. The camera did not interpret. It did not select. It did not impose the aesthetic preferences or theoretical commitments that inevitably shaped the illustrator's hand. It recorded what was there, with a fidelity no human could match and, more importantly, with an impartiality no human could achieve.

The argument was irresistible because it addressed the specific anxiety that had been building within the sciences for decades — the anxiety about the distorting influence of human subjectivity on the production of knowledge. The truth-to-nature illustrator was trusted because she was an expert, but her expertise was also the problem. Her trained eye saw what her training had prepared her to see. Her theoretical commitments shaped her perceptions. Her aesthetic conventions determined which features she emphasized and which she suppressed. The very qualities that made her a skilled illustrator — her judgment, her selectivity, her capacity for synthesis — were precisely the qualities that introduced bias into the representation.

Photography appeared to solve this problem by eliminating the problematic element entirely. The camera did not judge. It did not select. It did not synthesize. It produced an image through a causal process — the chemical reaction of light-sensitive compounds to photons reflected from the specimen — that was, in principle, independent of any human mind. The image was not drawn. It was caused. The causal chain ran from object through lens to photographic plate without passing through a consciousness at any point.

Daston and Galison identified the epistemic ideal that photography embodied as "mechanical objectivity" — the conviction that the path to reliable knowledge ran through the systematic suppression of the self. The observer should not interpret. The instrument should not select. The representation should be produced by a process as independent as possible of the particular human who operated it. The machine was trusted precisely because it could not exercise judgment. Judgment was the source of distortion. The solution was to build instruments that recorded without judging, that preserved without interpreting, that transmitted the world's signals without adding signals of their own.

The confidence that accompanied mechanical objectivity was, in many ways, more absolute than the confidence that had accompanied truth-to-nature, because the grounds for confidence had shifted from a human quality (expertise) to a mechanical property (causation). The illustrator's authority was acknowledged as fallible, because expertise is a human attribute and human attributes are imperfect. The photograph's authority appeared infallible, because the causal process that produced it was not subject to the errors of judgment, attention, or interpretation that plagued human observers. The photograph did not have a bad day. It did not bring its theoretical commitments to the darkroom. It simply registered what was there.

The implicit claim of reliability embedded in the photograph was therefore categorical in a way the illustration's claim had not been. The illustration said: "This is what the competent observer sees when she looks past the accidents to the essence." The claim acknowledged the mediating role of human judgment and staked its authority on the quality of that judgment. The photograph said: "This is what was there." The claim denied mediation altogether. It presented the image as a direct transcript of reality, produced by a process that admitted no interpretive intervention.

The claim was, of course, partially false — and partially false in ways that took decades to become fully visible.

The photograph encoded interpretive choices at every stage of its production: what to photograph, from what angle, at what distance, in what light, with what lens, at what exposure, developed by what chemical process, printed on what paper. These choices were real and consequential. A photograph of a microscopic specimen taken at one magnification told a different story than the same specimen at another. A photograph of a geological formation taken in morning light emphasized features that afternoon light suppressed. The development process itself introduced distortions — grain, contrast, tonal range — that were properties of the chemistry, not of the specimen.

But these choices were less visible than the illustrator's choices, for a reason that Daston's framework identifies as the key mechanism of epistemic overconfidence in every technology transition. The illustrator's interventions were evident in the strokes of her pen. They were visible on the surface of the image. A viewer could see that a human hand had produced this representation, and the visibility of the hand served as a constant reminder that the image was a product of judgment. The photographer's interventions were embedded in the apparatus — in the optics, the chemistry, the mechanical settings — rather than in visible marks of human intentionality. The interpretive choices were concealed by the technology, not because the technology was designed to conceal them, but because the form they took — focal length, exposure time, chemical sensitivity — was not recognizable as interpretation to users whose concept of interpretation had been formed by centuries of experience with manual representation.

This concealment is the mechanism that matters most for understanding the present moment. The new technology's biases are harder to see than the old technology's biases, not because the new technology has fewer biases but because its biases take a form that existing evaluative competencies are not equipped to detect. The practitioners who were expert at identifying distortions introduced by human illustrators had developed their expertise in a world where distortion was visible in pen strokes. When distortion migrated from pen strokes to exposure settings and chemical processes, the old expertise became inadequate, and new expertise had to be developed — a process that consumed decades.

Daston's attention to the material culture of knowledge production — the instruments, the recording media, the institutional formats — illuminates a dimension of this transition that purely conceptual analysis misses. The photograph was not simply a new idea about how to represent nature. It was a new material system — a network of instruments, chemicals, techniques, training programs, and institutional practices that together constituted the infrastructure through which photographic knowledge was produced, evaluated, and transmitted. The transition from illustration to photography was therefore not merely a change in representational philosophy. It was a transformation of the entire material culture of scientific representation — a transformation that required new skills, new institutions, new formats, new standards, and new criteria for evaluating the reliability of the resulting images.

The AI transition is producing an analogous transformation of material culture, though the specific elements are different. The training dataset is a new kind of epistemic object — a collection of human-produced texts so vast that no individual can survey it, so heterogeneous that no single disciplinary framework can organize it, and so consequential that its biases are reproduced in every output the system generates. The model architecture is a new kind of epistemic instrument — a mathematical structure that determines which patterns in the training data are relevant to a given query, through processes that are opaque to the user and incompletely understood even by the engineers who designed them. The chat interface is a new kind of epistemic format — a conversational structure that presents AI-generated text in a form indistinguishable from human-produced text, carrying all the implicit claims of reliability that conversational prose has accumulated over centuries of use.

Each of these material elements shapes the knowledge that the technology produces — what can be represented, how it is presented, what aspects of the underlying reality are emphasized, and what aspects are concealed. And each element introduces its own characteristic forms of distortion, its own confidence artifacts, its own patterns of warranted and unwarranted trust.

The training data is not a neutral sample of human knowledge. Daston, in a formulation that illuminates the problem with characteristic precision, has noted that the word "data" itself — from the Latin dare, to give — originally referred not to things found but to things given: the premises of an argument, the starting points that were granted rather than discovered. The transformation of data from the given to the found, from philosophical assumption to empirical observation, occurred over centuries and required the construction of elaborate systems of collection, classification, and standardization. AI training data inherits all the contingencies of these systems. It is shaped by which institutions produced and preserved texts, which languages those texts were written in, which formats they were recorded in, which economic structures determined what was worth digitizing. When an AI generates a response, it is not drawing on the sum of human knowledge. It is drawing on a historically specific, institutionally mediated, materially constrained subset of that knowledge, formatted for machine consumption.

The photograph did not replace the drawing overnight, and the transition was not a simple substitution of a superior technology for an inferior one. For decades, both technologies coexisted, and the most sophisticated scientific publications used both — photographs for the detailed record, illustrations for the interpretive synthesis. The practitioners who navigated the transition most successfully were those who understood the strengths and limitations of each technology, who could deploy each for the purposes to which it was suited, and who resisted the seductive clarity of treating either one as a transparent window onto reality.

The same dual competency will be required for the AI transition. The most valuable practitioners will be those who can use AI-generated knowledge where it is reliable, identify where it is not, and maintain the independent evaluative capacities that neither the technology nor its promotional culture will provide. This competency is not primarily technical. It is epistemological — a matter of understanding not just what the technology produces but how the process of production shapes what is produced.

The photograph taught a generation of scientists that mechanical reproduction was not the same as objective representation. The lesson took decades to learn. The AI transition is teaching an analogous lesson — that statistical fluency is not the same as genuine understanding — and the historical evidence offers no reason to expect the learning will be faster this time.

---

Chapter 3: The Confidence Artifact

Every knowledge-producing technology develops a repertoire of output features that inspire trust. These features are not incidental to the technology's operation. They are constitutive of its authority — the means by which the technology's outputs establish themselves as credible in the eyes of the communities that use them. The features are real, in the sense that they reflect genuine properties of the production process. They are also misleading, in the sense that they suggest a degree of reliability that the production process does not uniformly guarantee. The gap between the suggested reliability and the actual reliability is what makes these features confidence artifacts — output properties that inspire confidence in excess of epistemic warrant.

The concept requires careful delineation, because the phenomenon it names operates through a mechanism that is subtle and, in a precise sense, invisible. A confidence artifact is not a lie. The technology that produces it is not necessarily deceptive. The artifact is a structural feature of the relationship between the technology's outputs and the evaluative heuristics that its users bring to those outputs. It arises when a surface property of the output — a property that users have learned, through long experience, to associate with a depth property like accuracy or reliability — is present in cases where the depth property is absent. The surface property activates the learned association, the association generates trust, and the trust exceeds the warrant.

Consider the trajectory across successive knowledge-producing technologies. The scientific illustration of the eighteenth century derived its authority from the illustrator's demonstrable skill. The clean lines, the careful rendering of three-dimensional form, the precise depiction of surface texture — these features communicated mastery, and mastery was taken as a proxy for accuracy. The proxy was imperfect. An illustrator could render a specimen with breathtaking technical skill while systematically misrepresenting its morphology, because the aesthetic conventions of botanical illustration valued symmetry and regularity in ways that biological organisms do not reliably provide. But the correlation between skill and accuracy was strong enough to sustain the practice for generations.

The photograph shifted the grounds of confidence from human skill to mechanical causation. The sharpness of the photographic image, its wealth of detail, its capacity to render features below the threshold of unaided human perception — these properties communicated a different kind of authority. The authority was not personal (this expert is trustworthy) but procedural (this process is reliable). The photograph's sharpness said: precision. Its detail said: completeness. Its mechanical provenance said: impartiality. Each of these suggestions was partially warranted and partially misleading. A sharp photograph could be sharp and misleading simultaneously. A detailed image could be detailed about the wrong things. A mechanically produced representation could encode the biases of its apparatus as surely as a hand-drawn image encoded the biases of its artist.

The statistical table introduced yet another register of confidence. The decimal places, the confidence intervals, the mathematical notation — these features communicated quantitative rigor. A finding reported as "11.7 percent" carried an implicit claim of measurement precision that the same finding reported as "approximately twelve percent" did not. The decimal place was a confidence artifact: it suggested that the underlying measurement distinguished between 11.6 and 11.8, regardless of whether the data actually supported that level of discrimination. Theodore Porter's research on quantification in public policy demonstrated that numerical representations acquired authority not because they were more accurate than qualitative descriptions but because they appeared to be — because the format of quantitative presentation carried associations of objectivity that narrative formats did not.

The mechanism is consistent across these cases. A surface feature of the output — aesthetic skill, photographic sharpness, quantitative precision — activates a learned evaluative heuristic that associates the surface feature with a depth property (accuracy, completeness, rigor). The heuristic is not arbitrary. The associations it encodes reflect genuine correlations that hold in many cases. Skilled illustrators are often accurate. Sharp photographs are often informative. Precise measurements are often rigorous. But the correlations are imperfect, and the confidence artifact exploits the imperfection by presenting the surface feature as though it guaranteed the depth property it merely suggests.

AI's confidence artifact is its prose fluency — and it operates through the same mechanism as every previous confidence artifact, with two additional features that make it more powerful and more difficult to resist than any of its predecessors.

The first additional feature is ubiquity. Previous confidence artifacts were domain-specific. The photograph's sharpness was a confidence artifact only when photographic evidence was being evaluated. The statistical table's decimal places were confidence artifacts only when quantitative data was being assessed. Each artifact operated within a recognizable format that could, in principle, serve as a cue for critical evaluation — a reminder that this particular type of output required a particular kind of skepticism. AI's prose fluency is not domain-specific. It is present in every output the technology produces, across every subject, in every register. There is no domain of AI output where the confidence artifact is absent. The cue for skepticism would have to be: "I am reading text." The cue is too broad to function as a trigger for critical evaluation.

The second additional feature is mimicry. Previous confidence artifacts operated in formats that were identifiably different from human-produced knowledge. Photographs were recognizably photographs. Statistical tables were recognizably statistical. The format itself marked the output as the product of a specific technology, and the marking could serve as a prompt for the evaluative competencies appropriate to that technology. AI-generated text is not identifiably different from human-produced text. It operates in the same format — natural language prose — and reproduces the same features — sentence structure, paragraph organization, vocabulary, rhetorical strategies — that human experts produce. The mimicry is precise enough that the confidence artifact is, in many cases, literally indistinguishable from the genuine article.

These two features — ubiquity and mimicry — produce a confidence artifact of unprecedented scope and power. The artifact cannot be avoided by avoiding a particular format, because the format is natural language itself. It cannot be detected by recognizing a particular technological signature, because the signature mimics the very features that centuries of experience have taught readers to associate with human expertise.

Daston's work on the material culture of knowledge production suggests a further dimension of the problem that has received insufficient attention. The chat interface through which most users encounter AI-generated text is itself a confidence artifact. It presents the AI's output in a conversational format — question and response, the back-and-forth of intellectual exchange — that carries implicit claims about the nature of the interaction. Conversation is a format we associate with mutual understanding, with the give-and-take of minds engaging with shared problems. The chat interface borrows these associations, presenting the AI's statistical pattern-matching in the guise of intellectual partnership. The material format shapes the epistemic expectations, and the expectations shape the trust.

Segal describes this phenomenon from inside the experience when he writes about the discipline required to reject Claude's output "when it sounds better than it thinks" — when the prose quality exceeds the intellectual substance. The formulation is precise. "Sounds better than it thinks" identifies the exact mechanism of the confidence artifact: the surface property (how it sounds) exceeding the depth property (how it thinks). But the discipline required to maintain this distinction is formidable, because the distinction must be made against the grain of every evaluative habit that a lifetime of reading has installed.

When a human writes polished prose about constitutional law, the polish almost always indicates years of engagement with the subject. The correlation is strong enough to have been confirmed by virtually every reading experience the reader has ever had. AI breaks this correlation without breaking the heuristic it trained. The prose is polished. The engagement may or may not be there. The heuristic fires anyway, because heuristics do not update on exceptions. They update on the slow accumulation of counterexamples, processed through conscious reflection, over extended periods of experience.

This is what makes the confidence artifact so difficult to resist at the individual level. A reader can know, intellectually, that AI's fluency does not guarantee accuracy. She can remind herself, before each interaction, that the prose quality is not a reliable indicator of the underlying knowledge. And she will still, in her default mode of reading, extend more trust to polished prose than the epistemic warrant justifies, because the heuristic operates below the threshold of conscious evaluation. Awareness does not override the heuristic. Only the slow development of new evaluative habits can, and the development of those habits requires the kind of sustained, institutionally supported practice that the present moment has barely begun to provide.

Daston's analysis of how scientific communities eventually learned to see past the confidence artifacts of previous technologies suggests that the learning is possible but never fast. The evaluative competencies required to assess photographs critically — to distinguish between features of the specimen and features of the photographic process — took decades to develop. The competencies required to assess statistical findings critically — to recognize when decimal places masked imprecise measurements, when significance levels were artifacts of sample size — took longer still. The competencies required to assess AI-generated text critically — to evaluate substance independently of style, to detect errors that are embedded in fluent prose, to resist the automatic extension of trust that polished language triggers — will take at least as long, and the historical pattern suggests they will arrive later than the technology requires.

The confidence artifact is not a design flaw. It is a structural consequence of producing knowledge in a medium that carries epistemic associations. The photograph claimed objectivity because mechanical images carried associations of impartiality. The statistical table claimed rigor because numbers carried associations of precision. AI claims authority because polished prose carries associations of expertise. The associations are real. They reflect genuine correlations that hold across most of human communicative history. But the correlations are breaking, and the institutions that should be responding to the break are only beginning to recognize that it has occurred.

---

Chapter 4: Statistical Fluency as False Authority

In the spring of 2026, the International Science Council released a report documenting a phenomenon its authors had been observing with growing alarm across multiple disciplines: the citation of AI-generated summaries of research findings in place of the primary literature. Graduate students were citing AI-produced overviews of experimental results without having read the experiments. Policy analysts were incorporating AI-generated characterizations of statistical findings into briefing documents without consulting the underlying data. Journalists were using AI summaries of scientific consensus as though the summaries were themselves evidence rather than representations of evidence — representations whose accuracy depended entirely on the reliability of the summarizing process.

The phenomenon was not, in itself, surprising. Every knowledge-producing technology generates a temptation to substitute the technology's outputs for independent engagement with the underlying reality. The photograph tempted viewers to treat the image as equivalent to direct observation. The statistical table tempted readers to treat the numbers as equivalent to the phenomena they measured. The temptation is structural: the technology produces a convenient, accessible, polished summary of information that would otherwise require laborious, time-consuming, expertise-dependent direct engagement. The summary is easier. It is faster. And it looks authoritative.

What was surprising, and what the ISC report identified as genuinely new, was the completeness of the substitution. Previous technologies had produced summaries that were recognizably different from the originals they summarized. A photograph of a specimen was recognizably a photograph, not a specimen. A statistical summary of survey data was recognizably a statistical artifact, not raw observation. The format marked the summary as a summary — a representation that stood at a known distance from the reality it represented. AI-generated summaries of research findings, by contrast, were indistinguishable in format from human-produced summaries. They used the same vocabulary, the same rhetorical structures, the same conventions of qualification and citation. The distance between the summary and the underlying reality was invisible, because the format provided no cue that would prompt the reader to ask how far the summary had traveled from the source.

This invisibility is the operational consequence of the decorrelation between fluency and authority that Daston's framework predicts. The decorrelation has a structure that repays careful analysis, because understanding the structure is the precondition for developing the evaluative competencies that the transition demands.

Fluency, in the sense relevant to this analysis, is a surface property of communicative outputs. A text is fluent when its sentences parse grammatically, when its paragraphs follow a recognizable argumentative logic, when its vocabulary is appropriate to its subject and register. Fluency is genuinely valuable. It facilitates comprehension, enables communication across disciplinary boundaries, and makes complex ideas accessible to non-specialist audiences. The achievement of fluency in a domain is a real accomplishment, typically requiring years of sustained engagement with the subject.

Authority, in the relevant sense, is a depth property. A text has authority when its claims are accurate, when its characterizations of other positions are fair and precise, when its reasoning withstands critical scrutiny, when its qualifications are honest, when its conclusions follow from premises that are themselves well-supported. Authority is an epistemic property — a relationship between the text and the state of affairs it purports to describe.

These two properties have been correlated throughout the history of human knowledge production with a reliability so consistent that the correlation has become, for most readers, invisible. The correlation is built into the educational process itself: learning to write fluently about a subject requires the same years of engagement that produce genuine authority. The medical student who can write a polished case analysis has, in the process of acquiring that fluency, acquired the clinical knowledge that makes the analysis authoritative. The legal scholar who can produce an elegant brief has, through the same sustained engagement, developed the jurisprudential understanding that gives the brief its weight. Fluency is the visible trace of a process that simultaneously produces authority, and for this reason, fluency has served as a reliable proxy for authority for as long as humans have produced and evaluated written text.

Daston's historical research reveals an illuminating parallel in the early modern history of learned communication. The conventions of scholarly Latin — its elaborate syntax, its precise vocabulary, its complex system of citation and cross-reference — served as both a medium of communication and a credentialing mechanism. A scholar who could write in proper Latin had, by definition, received the extended training that produced scholarly competence. The language was the credential. It served simultaneously as a medium of expression and as a signal of the education that made the expression trustworthy. When vernacular languages began to displace Latin as the medium of scholarly communication in the seventeenth and eighteenth centuries, the credentialing function was disrupted: texts in French or English or German could be produced by anyone who spoke those languages, regardless of formal training. New credentialing mechanisms — university degrees, professional societies, peer-reviewed journals — had to be constructed to replace the one that the language shift had destroyed.

AI is producing an analogous disruption. The fluency that served as a natural credential — the visible trace of the sustained engagement that simultaneously produced expertise — can now be generated by a system that has not undergone the engagement. The credential has been decoupled from the process it used to certify. A text that reads like the product of deep expertise may be the product of statistical pattern-matching across a training corpus, and there is no feature of the text itself that allows the reader to distinguish between the two cases.

Daston, in her analysis of what she called the "moral economy" of science — the system of affect, trust, and obligation that governs knowledge-producing communities — showed that trust in scientific claims was never based solely on the content of those claims. It was based on a web of social, institutional, and material signals that together constituted the evidentiary infrastructure of the scientific community. The author's institutional affiliation, the journal in which the work appeared, the rigor of the peer review process, the reproducibility of the results, the coherence of the claims with established knowledge — all of these factors contributed to the credibility of the claim, and none of them was reducible to the claim's content alone.

AI-generated text bypasses this entire evidentiary infrastructure. It carries none of the institutional signals that the moral economy of science has developed over centuries to calibrate trust. It is not produced by an identifiable author whose reputation is at stake. It is not vetted by a peer review process whose standards are public and contestable. It is not embedded in an institutional context that provides accountability for errors. It is produced by a statistical process, presented in the same conversational format regardless of its accuracy, and evaluated by users who possess, in most cases, neither the domain knowledge nor the institutional support that reliable evaluation requires.

Segal's account of discovering the Deleuze error — the passage that "worked rhetorically" and "felt like insight" but mischaracterized the philosophical concept — is a case study in the consequences of the decorrelation. The passage passed the fluency test. It read like authoritative philosophical analysis. But it failed the authority test, and the failure was detectable only by someone who possessed independent knowledge of the philosophical literature — knowledge that the passage itself was meant to provide. The circularity is exact: the reader who needs the AI's summary lacks the expertise to evaluate it, and the reader who possesses the expertise does not need the summary.

This circularity was identified in Chapter 1 as a structural feature of every knowledge-producing technology's relationship to its users. But it takes a particularly acute form in the case of AI, because the technology is most frequently used in precisely the contexts where the circularity is most dangerous — contexts where the user is consulting the technology because she lacks the domain knowledge that would enable independent evaluation. The student who asks Claude to explain Deleuze is asking because she has not read Deleuze. The policy analyst who asks for a summary of climate research is asking because he has not read the primary literature. The journalist who asks for an overview of vaccine efficacy is asking because she lacks the epidemiological training that would allow her to assess the studies directly. In each case, the technology is consulted because the user cannot do what reliable evaluation requires — and the technology's fluent response provides no signal of its own reliability.

The consequences extend beyond individual cases of misinformation. The systematic substitution of AI-generated summaries for direct engagement with primary sources restructures the knowledge economy itself. When fluency can be generated without the process that produces authority, the market's willingness to subsidize that process erodes. Why invest years in mastering a body of literature when a machine can produce a fluent summary in seconds? The question is not rhetorical. It is being answered, in admissions offices and hiring committees and editorial boards, in ways that will shape the intellectual infrastructure of the coming generation.

Daston's analysis of the "thick rules" and "thin rules" that govern human rulemaking offers a framework for understanding why this erosion matters. In her 2022 book Rules: A Short History of What We Live By, Daston distinguished between "thick" rules — rules accompanied by copious contextual advice on their application, replete with examples, exceptions, and guidance for the exercise of judgment — and "thin" rules — rigid, predictive, requiring no discretion, designed to be executed mechanically. The history of rules, Daston showed, is a history of the gradual thinning of thick rules into thin ones, driven by the desire for predictability, efficiency, and the elimination of human variability. But behind every thin rule, she observed, is a thick rule cleaning up after it — a human exercising judgment about the cases where the thin rule fails, the exceptions it cannot accommodate, the contexts it was not designed to address.

AI-generated knowledge is thin knowledge. It is fluent, comprehensive, confident — and it requires thick knowledge to evaluate. It requires the judgment, the contextual awareness, the capacity for recognizing exceptions and anomalies, that only sustained engagement with a domain can produce. The decorrelation of fluency and authority means that thin knowledge is being produced at a rate that far exceeds the production of the thick knowledge needed to evaluate it. The ratio is worsening, because the same technology that accelerates the production of thin knowledge simultaneously reduces the incentive to invest in the thick knowledge that keeps it honest.

This is the structural problem that Daston's framework illuminates and that the present moment must confront. The solution is not to ban AI-generated knowledge, which would be neither feasible nor desirable. The solution is to build the institutional infrastructure — the evaluative mechanisms, the training programs, the professional standards — that restores the connection between the production of knowledge and the capacity to evaluate it. The connection has been broken by a technology that produces one without the other. Rebuilding it will require the kind of sustained institutional effort that Daston's historical research shows has been necessary after every major transition in the technology of knowledge production — and that has never, in the historical record, been achieved as quickly as the transition demanded.

Chapter 5: The Calibration Problem

In the early decades of the compound microscope's use, European naturalists reported observing structures in biological specimens that no one had seen before. The reports were received with excitement proportional to their novelty. Here was a technology that extended human perception beyond its previous limits, revealing features of the natural world that the unaided eye could never have detected. The structures were documented, cataloged, incorporated into taxonomic descriptions, cited in subsequent studies. They became part of the evidentiary record of the life sciences.

Some of the structures were real. Others were artifacts of the optical system — chromatic aberrations produced by imperfect lenses, diffraction patterns generated by the interaction of light with the aperture, distortions introduced by the preparation techniques that made specimens transparent enough to observe. The artifacts looked like biological structures. They appeared in the same visual field, at the same magnification, with the same apparent solidity and specificity. There was no mark on the image that distinguished an artifact from a genuine feature. Both presented themselves with identical visual authority.

The distinction could not be drawn by examining the microscope's output alone. Drawing it required independent methods — comparison with specimens examined by other techniques, theoretical analysis of the optical system's mathematical properties, controlled experiments that varied the instrument's parameters while holding the specimen constant. These methods took decades to develop, and during the intervening period, the evidentiary record of microscopy contained an unknown mixture of genuine observations and instrumental artifacts, with no reliable way to determine the proportion.

This is the calibration problem in its most general form. Calibration is the process by which users of a knowledge-producing technology learn to assess its outputs with accuracy proportionate to the outputs' actual reliability — to extend trust where trust is warranted and withhold it where it is not. A well-calibrated user of a microscope knows which features of the image correspond to features of the specimen and which are artifacts of the optics. A well-calibrated user of a statistical method knows which findings reflect genuine patterns in the data and which are artifacts of the analytical technique. A well-calibrated user of any knowledge-producing technology has developed, through sustained experience with the technology's characteristic modes of failure, an internalized model of the technology's reliability profile — a sense, built up through repeated encounters with the boundary between the technology's accurate and inaccurate outputs, of where to place confidence and where to withhold it.

Calibration develops through a specific mechanism: the encounter with identifiable error. The user produces or receives an output, compares it against some independent source of information, discovers a discrepancy, and updates her model of the technology's reliability. The discrepancy is the informational event. It reveals a case in which the technology was unreliable, and the specifics of the case — the type of error, the conditions under which it occurred, the features that distinguished the erroneous output from accurate outputs — contribute to the gradually accumulating profile of the technology's limitations. Over time, through repeated encounters with discrepancies of different types, the user develops a calibrated intuition: a sense, not always fully articulable but operationally reliable, of when the technology can be trusted and when it cannot.

The mechanism depends on two conditions that the history of scientific instrumentation reveals with uncomfortable clarity. The technology's errors must be detectable — the user must have access to independent information against which the output can be compared. And the errors must be identifiable — they must be distinguishable, in some way, from the technology's accurate outputs, so that the user can learn to recognize the markers that signal unreliability.

Lorraine Daston's historical research on the practices of scientific observation — the protocols, the institutional structures, the training regimes through which scientific communities taught their members to see — reveals that both conditions are harder to satisfy than they appear. The microscopist who encountered chromatic aberrations in her images had, in principle, access to independent information: she could compare the microscopic image with what she knew about the specimen from other methods of examination. But the whole point of the microscope was to reveal features invisible to other methods. The independent information was, by definition, unavailable for the very features the microscope was most valued for revealing. The technology was most trusted precisely in the domain where calibration was most difficult — where the outputs could not be compared against independent evidence because no independent evidence existed.

Daston identified this as a recurrent feature of the epistemological crises that accompany new instruments of observation. The telescope revealed features of celestial objects that could not be confirmed by naked-eye observation. The seismograph recorded patterns in the earth's vibrations that could not be felt by human senses. In each case, the instrument was valued for extending perception beyond its previous limits, and in each case, the extension created a domain of evidence for which no independent check existed. The instrument's most novel outputs were its least calibratable, because calibration requires comparison and comparison requires an independent standard.

AI exhibits this structure with particular intensity. The technology is most frequently consulted in domains where the user lacks independent knowledge — where the consultation is motivated precisely by the gap between what the user knows and what the technology can provide. A researcher who consults AI for a summary of a literature she has not read has no independent basis for evaluating the summary's accuracy. A student who asks AI to explain a concept she does not understand has no independent basis for assessing whether the explanation is correct. A professional who uses AI to draft an analysis in an area adjacent to but outside her expertise has no independent basis for determining whether the analysis reflects the current state of knowledge in that area.

In each of these cases — and they constitute the majority of AI use cases — the first condition for calibration is structurally unsatisfied. The user does not possess the independent knowledge that would enable her to detect errors in the AI's output. The errors, if they exist, are invisible not because they are subtle but because the user lacks the reference frame that would make them visible.

The second condition — the identifiability of errors — fares no better. The microscope's optical artifacts were, in principle, distinguishable from genuine features: they had characteristic forms (color fringes, concentric rings, particular types of geometric distortion) that a trained observer could learn to recognize. The photographic plate's chemical artifacts were similarly identifiable: grain patterns, tonal inversions, sensitivity variations across the spectrum. These artifacts had signatures — recurring features that marked them as products of the instrument rather than features of the specimen. The signatures were not immediately obvious. They required training and experience to recognize. But they existed, and their existence meant that calibration was, in principle, achievable through the accumulation of experience with the technology's characteristic modes of failure.

AI-generated text has no characteristic error signature. An AI-produced statement that is factually wrong is grammatically, syntactically, and rhetorically identical to an AI-produced statement that is factually correct. The confident assertion of a false claim is indistinguishable, in every surface feature, from the confident assertion of a true one. There is no visual artifact, no stylistic marker, no rhetorical tell that flags a particular passage as unreliable. The errors are not just difficult to detect. They are, in their surface presentation, invisible.

The combination produces a calibration problem qualitatively different from any that Daston's historical research has documented. Previous knowledge technologies presented calibration challenges that were difficult but soluble: the errors were hard to detect but not in principle undetectable, hard to identify but not in principle unidentifiable. AI presents a calibration challenge in which the standard mechanism — learning from encounters with identifiable errors — is structurally impaired, because the encounters do not reliably occur and the errors, when they do occur, carry no identifying marks.

Ludwik Fleck, the philosopher and microbiologist whose concept of the "thought style" Daston has identified as a precursor to her own historical epistemology, described the process by which a scientific community learns to see as a social and material achievement, not a natural capacity. The microscopist did not simply look through the lens and see. She was trained to see — taught by experienced practitioners which features of the image were significant and which were artifactual, which patterns indicated genuine structures and which were products of the preparation technique. The training was embodied, hands-on, extended over years. It could not be transmitted through textbooks alone. It required supervised practice with actual instruments, actual specimens, and actual errors — errors that the experienced practitioner could identify and the novice could learn from.

The Fleck framework illuminates why AI calibration is so much harder than calibration for previous technologies. The calibration of trust in microscopic images was achieved through a social process — communities of practitioners who shared instruments, compared observations, developed conventions for distinguishing artifact from genuine feature, and transmitted those conventions through training programs that extended over years. The calibration was institutional, not individual. It depended on the existence of a community with shared standards, shared access to the technology, and shared methods for evaluating the technology's outputs against independent evidence.

No analogous community exists for AI-generated knowledge. There is no established training program that teaches users to identify AI's characteristic modes of failure. There is no shared set of conventions for distinguishing reliable from unreliable AI output. There is no institutional mechanism that subjects AI-generated claims to the kind of systematic, expert-mediated evaluation that the peer review system provides for scientific claims. The calibration infrastructure that centuries of experience with other knowledge technologies have produced does not exist, in any remotely adequate form, for the technology that is being deployed most rapidly.

The historical pattern identified in Daston's research suggests that this infrastructure will eventually be built. Every previous knowledge technology has eventually generated the evaluative institutions appropriate to its characteristic modes of failure. Microscopy generated standardized testing protocols, calibration procedures, and training programs. Statistics generated meta-analysis, replication initiatives, and methodological review. Photography generated authentication standards, chain-of-custody protocols, and forensic examination techniques. Each institutional response was specific to the technology's particular calibration challenges, developed through decades of experience with the technology's particular modes of failure, and maintained by communities of practitioners whose expertise consisted precisely in the ability to distinguish between the technology's reliable and unreliable outputs.

But the historical pattern also shows, with a consistency that should alarm anyone paying attention, that the institutional infrastructure always arrives after the period when it is most needed. The calibration institutions are built in response to costly errors — errors that accumulate during the period before the institutions exist, errors that could have been prevented if the institutions had been in place, errors whose human cost constitutes the political motivation for building the institutions. The sequence is not logical but historical: first the technology, then the errors, then the institutions. The errors are the price of the gap between the technology's deployment and the development of the evaluative infrastructure it requires.

Segal's description of the Trivandrum engineer — the senior practitioner who lost architectural intuition after months of AI use because the formative struggle that had built the intuition was optimized away — is a case study in the calibration gap. The engineer's loss was not the result of a spectacular failure. It was the result of a gradual, invisible erosion of a capacity that had been built through friction and was not being maintained in the friction's absence. The engineer did not know what she had lost until the absence of the lost capacity manifested in diminished confidence. The calibration failure was prospective rather than retrospective — a loss of future capability rather than a present error — and therefore invisible to any evaluative mechanism designed to detect present errors.

This prospective dimension of the calibration problem has no clean historical precedent. Previous technologies produced errors that were, in principle, detectable at the time they occurred. A microscopic artifact was present in the image when the image was made. A statistical error was present in the analysis when the analysis was published. The errors were contemporaneous with the outputs that contained them. AI's most consequential calibration failures may be prospective — the erosion of capacities that will be needed in the future but whose absence is not detectable in the present. The developer who loses debugging intuition, the student who loses the capacity for independent analysis, the professional who loses the domain knowledge that would enable her to evaluate AI output — each of these losses is a calibration failure that manifests not as a present error but as a future incapacity.

The institutional response to this prospective dimension of the calibration problem requires a form of foresight that institutional design rarely achieves. The institutions must protect against harms that have not yet materialized, that may take years to become visible, and that will be detectable only by comparison with a counterfactual — the capacities that would have developed if the technology had not intervened. This is institutional design at its most difficult, and the historical record offers few examples of its successful execution. The examples that do exist — occupational safety regulations that protect against long-latency harms, environmental regulations that protect against cumulative damage — were typically enacted only after the harms had become visible, often decades after the exposure that produced them.

The calibration problem is not a problem that the market will solve. Markets optimize for present performance, and the calibration failures that matter most are failures of future capacity. The problem requires institutional intervention of a kind that runs against the grain of the technology's momentum — intervention that protects capacities whose value is not yet apparent, that maintains friction in a system optimized for its removal, that invests in evaluative infrastructure whose returns are measured in errors prevented rather than outputs produced.

Daston's research suggests that such intervention is achievable but rare, and that it typically arrives later than the harm it is designed to prevent. The question for the present moment is whether the historical pattern can be broken — whether the awareness that the pattern exists can accelerate the institutional response beyond the pace that precedent predicts. The awareness is necessary but not sufficient. The institutions must still be designed, funded, staffed, and maintained, and each of these steps encounters resistance from a technology whose momentum rewards speed over calibration and output over evaluation.

The microscope's calibration problem was eventually solved. The solution took decades. It required the construction of an entire institutional apparatus — training programs, testing protocols, standardized procedures, communities of expert practitioners — that did not exist when the technology was first deployed. The period between the deployment and the institutional response was the period of costly errors, and the errors were borne by the users who trusted the technology most completely at the moment when they understood its limitations least.

The AI calibration problem is harder. The errors are less visible. The institutional response is less advanced. And the technology is being deployed at a speed that no previous knowledge technology has approached. The gap between deployment and institutional response is not closing. The historical evidence provides no reason to expect it will close on its own.

---

Chapter 6: What Each Era's Users Had to Learn

The history of knowledge-producing technologies is, among other things, a history of lessons — specific, hard-won, institutionally codified lessons about the particular ways in which particular technologies deceive. Each era's users inherited the lessons of the previous era and had to learn new ones, because each new technology introduced new forms of distortion that the previous era's lessons did not address. The lessons were not general. They were specific to the technology's characteristic modes of failure, and they could not be derived from abstract epistemological principles alone. They had to be learned through experience with the technology itself, through the accumulation of cases in which the technology's implicit claims of reliability were tested against reality and found wanting.

The accumulation was slow, contested, and institutionally mediated. It did not happen automatically. It happened because specific people — scientists, methodologists, critics, reformers — identified specific errors, documented them, communicated them to their professional communities, and argued for the adoption of specific practices designed to prevent their recurrence. Each lesson had advocates and opponents. Each practice had to be justified against the resistance of practitioners who had invested their careers in the methods the new practices were designed to correct. The history of epistemological learning is a history of institutional politics as much as a history of intellectual progress.

Users of truth-to-nature scientific illustrations had to learn that the illustrator's aesthetic judgment was not epistemologically transparent — that the decisions about what counted as essential and what counted as accidental were genuine decisions, shaped by theoretical commitments and aesthetic conventions that varied across schools, traditions, and historical periods. The lesson was codified in practices that made the illustrator's interpretive role visible: standardized conventions for distinguishing observed from inferred features, the use of dotted lines to indicate conjectured structures, the inclusion of multiple views to prevent a single perspective from determining the representation. These practices did not eliminate the illustrator's interpretive role. They disciplined it, by making the interpretive choices legible to the reader and therefore available for critical evaluation.

The process was neither smooth nor rapid. Daston's research documents the resistance that met proposals for standardized representational conventions. Illustrators defended their professional autonomy. Patrons defended their aesthetic preferences. Scientific societies debated the appropriate balance between interpretive synthesis and mechanical fidelity. The debates extended over decades and were resolved differently in different disciplines, reflecting the different epistemic cultures of botany, anatomy, mineralogy, and zoology. The lesson was not learned once and universally applied. It was learned piecemeal, institution by institution, discipline by discipline, through a process of negotiation that was as much social and political as it was epistemological.

Users of photographs had to learn a different lesson: that the mechanical process did not eliminate interpretation but concealed it. Framing, exposure, magnification, focal plane, development chemistry — each of these constituted an interpretive choice, but the choices were embedded in the apparatus rather than visible in the marks of a human hand. The concealment made the lesson harder to learn, because the evaluative competencies that had been developed for identifying interpretive choices in illustrations were adapted to a world where those choices were visible on the surface of the image. When the choices migrated from the surface to the apparatus, the old competencies became inadequate.

The lesson was learned through what might be called epistemic scandals — episodes in which photographic evidence was revealed to be misleading in ways that the technology's implicit claim of objectivity had concealed. In astronomy, features recorded on photographic plates were shown to be artifacts of the emulsion's chemical properties rather than features of the celestial objects being observed. Entire programs of astronomical research had to be reassessed when the distinction between genuine features and photographic artifacts was finally established. In forensics, photographic evidence presented to juries was shown to carry implications that the images themselves did not support — implications created by framing choices, lighting conditions, and the selective inclusion or exclusion of contextual information. In anthropology, photographs presented as objective records of cultural practices were revealed to reflect the photographer's preconceptions about the cultures being documented — preconceptions that were invisible in the images themselves but legible to later observers who could identify the interpretive conventions that had shaped them.

Each scandal motivated institutional reform. Standards for documenting the technical parameters of scientific photographs were developed. Protocols for the authentication of photographic evidence in legal proceedings were established. Training programs for the critical interpretation of photographic images were created. The reforms addressed specific failure modes revealed by specific scandals, and they were adopted through the same contentious institutional processes that characterized the earlier reforms in scientific illustration.

Users of statistical methods had to learn a lesson more complex still: that the mathematical apparatus of statistical analysis carried an implicit claim of objectivity that was itself a confidence artifact. Numbers, unlike prose or images, do not argue. They present. The statistical table says "here are the results" in a format that appears to exclude interpretation — that appears to be a direct record of measurement, unmediated by human judgment. The lesson that had to be learned was that the appearance of interpretive absence concealed interpretive choices at every stage: the selection of variables, the construction of categories, the choice of statistical tests, the criteria for significance, the methods of data collection, the handling of missing data, the decisions about what to measure and what to ignore.

The lesson was codified in the methodological canon that now governs quantitative research across the social and behavioral sciences: the requirement to pre-register hypotheses, to report methods in sufficient detail for replication, to justify category choices, to disclose sample characteristics, to specify significance criteria in advance, to acknowledge limitations. These practices constitute a sophisticated institutional response to a specific calibration challenge — the challenge of evaluating statistical claims when the format of their presentation (numbers, tables, mathematical notation) carries confidence-inspiring associations that may exceed the rigor of the underlying process.

Daston's analysis of what she termed the "moral economy" of scientific practice reveals that these methodological reforms were never purely technical. They were also moral — they reflected and enforced specific values about transparency, accountability, and the obligations that knowledge producers owe to their communities. The requirement to report methods is not merely a technical standard. It is a moral commitment to transparency, a recognition that the reader has the right to evaluate the process that produced the claims she is being asked to accept. The requirement to acknowledge limitations is not merely a rhetorical convention. It is a moral commitment to honesty about the boundaries of one's knowledge, a recognition that the claims of expertise carry obligations as well as privileges.

The moral dimension of epistemological learning is particularly relevant to the AI transition, because the technology's most consequential effects operate in a domain that is simultaneously epistemological and moral — the domain of trust. Trust is not merely a cognitive state (the belief that a source is reliable). It is a social relationship, embedded in institutional structures, governed by norms of reciprocity and accountability, and sustained by the expectation that the trusted party will honor the obligations that trust imposes. When a reader trusts a scientific paper, the trust is underwritten by an institutional infrastructure — peer review, replication, professional accountability — that imposes costs on betrayal and rewards fidelity. When a reader trusts an AI-generated text, no comparable infrastructure exists. The trust is underwritten by nothing beyond the text's surface properties — its fluency, its confidence, its structural polish.

What must the users of AI learn? The lesson has the same structure as every previous lesson but content that is, in important respects, unprecedented.

The structural similarity: AI's outputs carry an implicit claim of reliability that exceeds the actual reliability, and the gap between the claim and the reality must be made visible through evaluative practices that the technology itself does not provide.

The unprecedented content: the feature that must be distrusted — prose fluency — is the feature that every previous knowledge technology has taught users to trust. The heuristic that must be overridden is the most deeply trained evaluative habit in the history of literate culture. And the override must be selective: it must apply to AI-generated text but not to human-generated text, in a context where the two are becoming increasingly indistinguishable.

Daston, in a 2022 conversation about her book Rules, offered a formulation that bears directly on the AI lesson: "Behind every thin rule is a thick rule cleaning up after it." The observation, intended as a comment on the history of algorithmic governance, applies with full force to the evaluative challenge that AI poses. AI-generated knowledge is thin knowledge — fluent, comprehensive, confident, produced by thin rules (statistical pattern-matching across vast corpora) that require no discretion, no contextual judgment, no understanding of the domain in which they operate. Evaluating that knowledge requires thick knowledge — the kind of contextual, domain-specific, judgment-dependent understanding that only sustained engagement with a field can produce. The thin rule generates the output. The thick rule determines whether the output is worth trusting.

The historical pattern suggests that the AI lesson will be learned, but that the learning will require the same institutional apparatus that every previous lesson has required: training programs that develop the relevant evaluative competencies, professional standards that codify the lessons of experience, feedback mechanisms that make the technology's errors visible to its users, and institutional cultures that value the exercise of judgment over the consumption of outputs.

The pattern also suggests that the apparatus will be built slowly, will arrive later than the technology requires, and will be adequate to the most common modes of failure while remaining vulnerable to the rarer and more subtle ones. This is not a prediction of failure. It is a prediction of difficulty — difficulty of a kind that the historical record shows has been navigated before, at costs that were real and significant, through institutional efforts that were imperfect but genuinely useful.

The lesson is the same lesson it has always been, in different dress: the technology is powerful, its outputs are useful, and its implicit claims of reliability must be evaluated by human judgment that the technology itself cannot supply. The institutional structures that support that judgment are the dams in the river — not barriers to the flow but structures that direct the flow toward habitable ground. They must be built. They must be maintained. And they must be built by people who have learned the specific lesson that this specific technology, in this specific moment, demands.

---

Chapter 7: The Period of Costly Errors

Every knowledge technology transition includes a period of demonstrable harm — a phase during which the technology's unrecognized limitations produce consequences that range from the embarrassing to the catastrophic. The period is not an anomaly or a failure of implementation. It is a structural feature of the relationship between a technology's deployment and the development of the evaluative infrastructure that its reliable use requires. The deployment moves at the speed of adoption. The infrastructure moves at the speed of institutional deliberation. The gap between the two is where the costly errors occur.

The structure of the period is characteristic and identifiable across different technologies and different centuries. The errors concentrate in domains where the technology's confidence artifacts are most persuasive, where the users are most dependent on the technology's outputs, and where the independent knowledge that would enable error detection is scarcest. The costs are borne not by the technology's producers — who profit from adoption regardless of calibration — but by the users who make decisions on the basis of outputs they have not yet learned to evaluate, and by the downstream communities affected by those decisions.

The photographic period of costly errors extended from the 1840s through the early decades of the twentieth century and produced its most consequential damage in precisely the domains where the photograph's implicit claim of objectivity was most persuasive. In forensic practice, photographic evidence was admitted to courtrooms with an authority that the technology's actual evidentiary value did not support. The photograph of a crime scene carried an implicit claim — "this is what was there" — that juries received as equivalent to direct testimony from an unimpeachable witness. The claim was powerful because it activated the heuristic of mechanical objectivity: the camera could not lie, could not be mistaken, could not be influenced by the biases that compromised human testimony.

But the camera could frame. It could include some elements and exclude others. It could be positioned to create spatial relationships that did not exist in the actual scene. It could be lit to emphasize features that would have been unremarkable under ambient conditions. And the photograph could be presented — selected, sequenced, captioned, contextualized — in ways that created narrative implications beyond what the image itself contained. The forensic errors that resulted were not errors of fabrication. They were errors of misplaced trust — the extension of confidence to a technology whose outputs were treated as transparent when they were, in fact, mediated by a series of choices that the format concealed.

The statistical period of costly errors extended across the twentieth century and produced its most visible crisis in the reproducibility failures that erupted publicly in the 2010s, though the underlying problems had been accumulating for decades. Researchers across the social and behavioral sciences had been using significance testing as a proxy for truth — treating a p-value below the conventional threshold as evidence that a finding was real, without adequately accounting for the many ways in which statistical significance could be produced by artifacts of the research process: publication bias that suppressed null results, researcher degrees of freedom that allowed post hoc selection of analyses, sample sizes too small to support the precision that the statistics implied, and the quiet, endemic practice of adjusting hypotheses after seeing the data.

The reproducibility crisis revealed that the published literature in several major fields contained a substantial proportion of findings that could not be replicated — findings that had been accepted, cited, incorporated into textbooks and policy recommendations, on the strength of statistical claims whose implicit authority exceeded their actual reliability. The costs were real and measurable: clinical interventions based on unreplicable findings, educational practices based on studies that dissolved under replication, public health recommendations supported by evidence that was, in retrospect, an artifact of the methods that produced it rather than a feature of the phenomena it purported to describe.

Daston's framework illuminates why the costly errors accumulate before rather than after the institutional response. The answer is not that institutions are slow, though they are. The answer is that the errors serve as the political motivation for building the institutions. The calibration infrastructure is expensive — it requires funding, staffing, political will, and the cooperation of practitioners who may have professional reasons to resist scrutiny of their methods. The motivation for incurring these costs typically comes not from abstract epistemological argument but from the accumulation of concrete harms that make the case for institutional reform undeniable. The errors are not merely the prelude to the institutional response. They are its precondition. The institutions are built because the errors have made the need for them visible.

This sequence — first the technology, then the errors, then the institutions — is the historical regularity that should most concern anyone tracking the AI transition. The sequence is not logically necessary. It is historically persistent. And the difference matters, because historical persistence can, in principle, be broken by foresight — by the recognition that the pattern exists and the deliberate decision to build the institutions before the errors accumulate to the point where they become the political catalyst for doing so.

The AI period of costly errors has identifiable structural features that distinguish it from previous periods while sharing their underlying logic. Three features deserve attention.

The first is the invisibility of the errors at the point of occurrence. Previous technologies produced errors that were, at least in principle, detectable at the time they were made. A misleading photograph existed as a physical artifact that could be re-examined. A flawed statistical analysis was a published document whose methods could be scrutinized. The errors were present in the outputs, waiting to be discovered by anyone with the expertise and the motivation to look. AI's errors include a category that has no clean precedent: the prospective erosion of capacities that the technology's use displaces. The student who uses AI to produce analyses without performing the analytical work herself does not commit an error in any immediately detectable sense. The analysis may be accurate. The student may receive a good grade. The error is prospective — a failure to develop the analytical capacities that the exercise was designed to build — and it will not become visible until the capacities are needed and found to be absent.

Segal describes this prospective dimension when he recounts the experience of the Trivandrum engineer who lost architectural intuition after months of AI-assisted work. The loss was invisible at the time it occurred. The engineer's outputs were better than ever — faster, more comprehensive, more polished. The diminishment was not in the outputs but in the producer, and it was detectable only months later, when the engineer discovered that she was making decisions with less confidence and could not explain why. The costly error was not a wrong answer. It was the gradual attrition of the capacity to generate right answers independently — a capacity whose absence would not be felt until the moment it was needed.

The second distinguishing feature is the distribution of the costs. In previous technology transitions, the costs of error fell most heavily on specific communities — the jurors who were misled by photographic evidence, the patients who received treatments based on unreplicable findings, the communities whose environmental exposures were underestimated by flawed models. The costs were concentrated and therefore, eventually, politically mobilizable. The communities that bore the costs could organize, advocate, and demand the institutional reforms that would protect them.

AI's costs are diffuse. The erosion of evaluative capacity is distributed across every domain in which AI is used, across every educational institution in which AI-assisted work is submitted, across every professional context in which AI-generated analysis is accepted without independent verification. The diffusion makes the costs harder to aggregate, harder to attribute, and harder to mobilize politically. No single community bears enough of the cost to generate the concentrated political pressure that has historically been the catalyst for institutional reform.

The third distinguishing feature is the speed of accumulation. Previous technologies produced their costly errors over decades, giving societies time — inadequate time, but time nonetheless — to observe the errors, document them, debate their significance, and develop institutional responses. Photography's forensic errors accumulated over half a century before the development of adequate authentication standards. The reproducibility crisis in social science built over decades before the replication initiatives of the 2010s. AI is producing its costly errors at the speed of digital adoption — across millions of users, in thousands of domains, at a rate that compresses into months what previous technologies distributed across decades.

Daston's research on the relationship between scientific crises and institutional reform suggests that the speed of accumulation matters enormously for the quality of the institutional response. Reforms developed in response to slowly accumulating evidence tend to be more carefully designed, more thoroughly debated, and more responsive to the specific features of the problem than reforms enacted in response to sudden crises. The slow accumulation allows time for the development of expertise — the specific knowledge of the technology's failure modes that the design of effective calibration institutions requires. Rapid accumulation produces crisis-driven reforms that address symptoms rather than structures, that respond to the most visible errors rather than the most consequential ones, and that are often superseded by the next wave of technological change before they have been fully implemented.

The prospect for the AI transition, evaluated against this historical evidence, is concerning. The errors are accumulating rapidly. The institutional response is developing slowly. The gap between the two is widening, as Segal observes, and the people in the gap are adapting in real time, without institutional support, through individual trial and error that produces local learning but not the systematic, transmissible, institutionally codified calibration that the situation requires.

The historical evidence does not, however, support despair. The pattern shows that the institutional response eventually arrives and that it works — imperfectly, belatedly, but genuinely. The evaluative practices and institutional standards that emerge from each technology's period of costly errors do reduce the gap between the technology's implicit claims and its actual reliability. The reduction is never complete. The institutions are never fully adequate. But they are better than nothing, and the difference between having them and not having them is the difference between a community that learns from its encounters with a powerful technology and a community that repeats the same errors indefinitely.

Daston, in a 2025 interview following her receipt of the Balzan Prize, remarked that "objectivity is not just a way of proceeding. It's an ethos which you dedicate yourself to, if you're a scientist." The observation applies beyond science. Calibrated trust — the disciplined practice of extending confidence in proportion to evidence, of evaluating claims independently of their surface presentation, of maintaining the capacity for independent judgment in the face of technologies that make judgment appear unnecessary — is an ethos, not merely a technique. It requires institutional support. It requires community. It requires the kind of sustained, deliberate, collectively maintained commitment that no individual can sustain alone and no technology will provide.

Building that commitment is the institutional project of the present moment. The period of costly errors has begun. The question that history poses, with the dispassionate persistence of a pattern that has repeated across centuries, is not whether the institutions will eventually be built. It is whether they will be built in time to reduce the cost of the transition for the people who are bearing it now.

---

Chapter 8: Institutional Adaptation and the Architecture of Calibrated Trust

The calibration of trust in a knowledge-producing technology is not achieved by individuals, however disciplined. It is achieved by institutions — by the sustained, collective, formally organized efforts of communities that develop shared standards for evaluating the technology's outputs, shared methods for detecting its characteristic errors, and shared practices for transmitting the resulting evaluative competencies to new users. Individual discipline is necessary. It is not sufficient. The historical record is unambiguous on this point: every technology whose users eventually developed calibrated trust did so through institutional mechanisms, and the quality of the calibration was determined by the quality of the institutions.

Daston's research on the institutional history of scientific objectivity reveals a pattern of institutional development that proceeds through three distinguishable phases, though in practice the phases overlap and interact in ways that resist clean periodization.

The first phase is recognition — the period during which the technology's characteristic limitations are identified by practitioners who encounter them in the course of their work. The recognition is initially individual, scattered, and often inarticulate. A microscopist notices that a particular feature appears only in images produced with a particular lens. A statistician notices that a particular finding disappears when the sample is stratified differently. A photographer notices that a particular feature of a specimen is visible only at particular exposures and suspects that the feature is an artifact of the photographic process rather than a property of the specimen. Each of these observations is a data point in the emerging picture of the technology's limitations, but the data points are uncoordinated, unpublished, and often unrecognized by the observers themselves as instances of a systematic pattern.

The second phase is articulation — the period during which the scattered observations of the recognition phase are collected, compared, organized, and given conceptual form. Articulation requires a vocabulary — a set of concepts that enable practitioners to name what they have observed and to communicate it to others with sufficient precision for the observations to be recognized as instances of the same phenomenon. The vocabulary of "chromatic aberration," "spherical distortion," and "diffraction artifact" gave microscopists the conceptual tools to describe what they had been observing individually and to recognize that their individual observations were manifestations of systematic, predictable, instrument-dependent phenomena. The vocabulary transformed scattered observations into a structured understanding of the technology's limitations.

The third phase is institutionalization — the period during which the articulated understanding is codified into standards, practices, protocols, and training programs that can be transmitted to new users without requiring each one to rediscover the technology's limitations through personal experience. Institutionalization is the mechanism that converts individual learning into collective knowledge, personal discipline into professional standard, episodic insight into systematic practice.

The three phases do not proceed in a clean temporal sequence. Recognition continues throughout the process, because new failure modes are discovered as the technology evolves. Articulation is revised in light of institutionalization, because the attempt to codify understanding into practice reveals gaps and imprecisions in the conceptual framework. Institutionalization is challenged by new developments that render existing standards inadequate and require the cycle to begin again. The process is iterative, open-ended, and never complete.

For AI-generated knowledge, the current moment sits at the overlap between recognition and the earliest stages of articulation. Individual practitioners are encountering the technology's limitations through personal experience — discovering cases where AI output is misleading, unreliable, or subtly wrong in ways that the output's surface quality conceals. Some of these encounters are being documented and shared, through blog posts, conference presentations, informal professional networks, and published studies like the Berkeley research that Segal discusses in The Orange Pill. But the documentation is fragmented, the sharing is unsystematic, and the conceptual vocabulary is still underdeveloped.

The concept of the "confidence artifact," as developed in this book, is an attempt to contribute to the articulation phase — to provide a conceptual tool that enables practitioners to name a phenomenon they have been encountering individually and to recognize their encounters as instances of a systematic, predictable, technology-dependent pattern. Other concepts — "hallucination," "confabulation," "sycophancy" — have emerged from the AI research community itself, reflecting the engineering perspective from which those concepts were developed. The historian's vocabulary is different because the historian's questions are different: not "how can the technology be improved?" but "how can users develop the evaluative competencies that the technology's reliable use requires?"

The vocabulary matters because it shapes the institutional response. If the problem is understood as "hallucination" — a technical defect to be corrected through improved model architecture — the institutional response is technical: better models, better training data, better alignment procedures. If the problem is understood as "confidence artifact" — a structural feature of the relationship between any knowledge-producing technology and its users — the institutional response is institutional: better evaluative practices, better training programs, better standards for the critical assessment of technology-mediated knowledge. The framing determines the response, and the adequacy of the response depends on the accuracy of the framing.

Daston's work on the moral economy of scientific communities suggests a dimension of the institutional challenge that purely technical analyses miss. Trust in knowledge claims is never based solely on the claims' content. It is sustained by a web of social relationships, institutional structures, and moral norms that together constitute the infrastructure of epistemic accountability. A scientific paper is trusted not only because its methods are sound but because it is embedded in a system of accountability: the author's name is attached, the peer reviewers have vouched for its quality, the journal's reputation is at stake, the methods are reported in sufficient detail for replication, and the norms of the scientific community impose real costs — professional ostracism, retraction, reputational damage — on practitioners who violate the community's standards of honesty and rigor.

AI-generated text is embedded in no comparable system. It carries no author's reputation. It has not been vetted by peers. It is not embedded in an institutional context that provides accountability for errors. The text arrives in the user's chat window bearing only its surface properties — its fluency, its confidence, its structural polish — and the user's evaluation of the text is supported by nothing beyond whatever individual expertise and critical discipline the user happens to bring to the interaction.

Building a culture of calibrated trust for AI-generated knowledge requires, at minimum, three institutional elements that the historical pattern identifies as necessary for the calibration of any knowledge-producing technology.

The first element is feedback mechanisms — structures that make the technology's errors visible to its users, enabling the learning-from-error process on which calibration depends. For previous technologies, feedback mechanisms included controlled experiments that tested the technology's outputs against independent standards, comparative studies that evaluated the technology's reliability across different domains and conditions, and systematic error databases that accumulated and categorized the technology's characteristic failure modes. For AI, analogous mechanisms might include automated fact-checking systems that flag outputs where the model's confidence is inconsistent with the available evidence, expert annotation systems that allow domain specialists to mark AI outputs as accurate or inaccurate, and systematic benchmarks that evaluate the technology's reliability across domains, question types, and levels of complexity.

The development of such mechanisms is technically feasible. It is not, however, technically inevitable. The mechanisms must be designed, funded, maintained, and integrated into the workflows of the communities that use AI. Each of these steps requires institutional will — the collective decision to invest in evaluative infrastructure whose benefits are measured in errors prevented rather than outputs produced. The investment runs against the grain of the technology's promotional culture, which emphasizes capability rather than limitation, speed rather than care, output rather than evaluation.

The second element is professional standards — codified practices that specify how AI-generated knowledge should be evaluated, disclosed, and used within specific professional contexts. The analogy here is to the methodological standards that govern quantitative research (pre-registration, open data, replication requirements) and the evidentiary standards that govern forensic practice (chain of custody, authentication protocols, expert testimony requirements). Professional standards for AI use would specify, among other things, the circumstances under which AI-generated content must be disclosed, the methods by which AI outputs must be verified before they are relied upon, the documentation that must accompany AI-assisted work, and the training that practitioners must complete before incorporating AI into professional practice.

Some professional organizations have begun to develop such standards. Medical journals have issued guidelines for the disclosure of AI assistance in the preparation of submitted manuscripts. Legal bar associations have begun to address the use of AI in legal research and brief preparation. Educational institutions have developed — often hastily and unevenly — policies governing the use of AI by students. These early efforts are important as precedents, but they are typically general in formulation, uneven in implementation, and developed without the systematic understanding of the technology's failure modes that effective standards require.

The third element is educational programs — structured training that develops the evaluative competencies on which calibrated trust depends. The most important of these competencies is the capacity to evaluate content independently of presentation — to assess the substance of claims without being influenced by the rhetorical authority of the prose in which they are expressed. This competency is not currently a standard component of education at any level. It requires explicit cultivation, sustained practice, and institutional support.

Daston's research on the training of scientific observers — the protocols through which scientific communities taught their members to see accurately, to distinguish between genuine features and artifacts, to calibrate their trust in the instruments they used — reveals that the development of evaluative competencies is always a social process. The microscopist learned to see not by reading a manual but by working alongside experienced practitioners who could identify artifacts the novice could not yet recognize, who could explain the mechanisms that produced those artifacts, and who could model the evaluative discipline that calibrated use required. The training was hands-on, extended, and embedded in a community of practice whose shared standards provided the framework within which individual skill developed.

The educational programs needed for AI will require analogous features: hands-on experience with the technology's failure modes, guided by practitioners who understand both the technology's capabilities and its characteristic limitations; explicit instruction in the distinction between fluency and accuracy, between confidence and reliability, between comprehensive-seeming answers and genuinely comprehensive understanding; and sustained practice in the discipline of independent evaluation — the habit of checking AI-generated claims against primary sources, of questioning confident assertions, of maintaining the intellectual self-reliance that the technology's convenience constantly erodes.

The three elements must work in combination. Feedback mechanisms without professional standards produce individual learning that is not aggregated into collective knowledge. Professional standards without educational programs produce rules that practitioners do not understand well enough to apply wisely. Educational programs without feedback mechanisms produce competencies that are not grounded in systematic evidence about the technology's actual reliability profile. The combination is the institutional ecosystem that calibrated trust requires, and the ecosystem must be cultivated deliberately, because it will not emerge spontaneously from the technology's adoption.

Daston argued, in her analysis of the shift from "truth-to-nature" to "mechanical objectivity" and then to "trained judgment" as successive ideals of scientific representation, that each transition required not merely new techniques but new epistemic virtues — new conceptions of what intellectual honesty demanded, new standards for what counted as responsible knowledge production, new norms for the relationship between the observer and the observed. The transition to AI-mediated knowledge production requires an analogous reconception. The relevant virtue is neither the expert judgment of truth-to-nature (trust the trained eye) nor the self-suppression of mechanical objectivity (trust the machine) nor the interpretive expertise of trained judgment (trust the trained interpreter of the machine's output). It is something closer to what might be called calibrated independence — the capacity to use the technology's outputs as inputs to one's own judgment rather than as substitutes for it, to maintain the intellectual self-reliance that enables independent evaluation even as the technology makes that self-reliance feel unnecessary.

The virtue is demanding. It requires resisting a technology that is designed for ease, maintaining friction in a system optimized for its removal, investing in capacities whose value is not immediately apparent. The institutions that support it must be designed with equal care — designed not to prevent the technology's use but to ensure that its use does not erode the human capacities on which its reliable use depends.

The historical evidence suggests that such institutions are achievable, that they genuinely reduce the gap between a technology's implicit claims and its actual reliability, and that the communities that build them navigate the transition more successfully than those that do not. The evidence also suggests that the institutions are always built later than needed, maintained less diligently than required, and adapted more slowly than the technology's evolution demands. The pattern is not encouraging, but it is not hopeless. The institutions work, when they exist. The question that the present moment is answering, with consequences that will be visible for generations, is whether they will be built in time.

Chapter 9: The Historical Pattern and What It Predicts

The pattern is now visible in its entirety, and its regularity across five centuries of knowledge-producing technology warrants the provisional confidence of a historical generalization — not a law, which would imply necessity, but a pattern, which implies that the same structural conditions have produced the same structural outcomes with sufficient consistency to inform expectation about the present case.

The pattern has five stages. They are not speculative. They are the retrospective organization of documented transitions — from hand-drawn illustration to photography, from qualitative description to statistical analysis, from analog measurement to computational modeling — each of which produced the same sequence of epistemic events in the same order, with variations in tempo and severity but not in structure.

The first stage is the threshold — the moment at which a new technology crosses a capability boundary that makes the previous method not merely less efficient but categorically different in kind. Writing did not make oral memory slightly less useful. It made memory structurally unnecessary for the long-distance transmission of complex knowledge. Photography did not make illustration slightly slower by comparison. It made the hand-drawn scientific image an act of interpretive choice rather than a method of objective recording. The threshold is not a quantitative improvement. It is a qualitative rupture — a moment after which the terms of the epistemological conversation have changed so fundamentally that the previous terms no longer apply.

AI crossed this threshold when it began producing text that was indistinguishable, in its surface properties, from the text produced by human domain experts. The capability that crossed the threshold was not speed or volume, though both are relevant. It was mimicry — the capacity to reproduce the surface markers of expertise (fluency, structure, confidence, appropriate register) without the underlying process that those markers had previously indicated. The mimicry is what makes the current transition categorically different from the computerization of calculation or the digitization of information retrieval. Those earlier transitions accelerated existing processes. This one counterfeits the signals by which expertise has been recognized for as long as human beings have communicated in writing.

The second stage is exhilaration — the period during which the technology's early users experience the genuine expansion of capability that the threshold represents. The exhilaration is not false. It reflects a real increase in what can be accomplished, a real reduction in the barriers between intention and execution, a real democratization of capacities that were previously confined to specialists. The first users of photography experienced it. The first users of statistical methods experienced it. Segal describes it with characteristic honesty when he recounts the weeks in Trivandrum during which his engineering team discovered that each member could accomplish what had previously required a full team — and the description rings true against the historical evidence, because the exhilaration is always genuine, always based on a real expansion, always warranted by the technology's actual capabilities.

The third stage is resistance — the protest of practitioners whose expertise the new technology has devalued. The resistance, as argued through the Luddite history in The Orange Pill, is grounded in real loss. The bards who memorized epic verse lost their livelihood when writing made memory dispensable. The scientific illustrators lost their authority when photography made manual rendering appear subjective. The statisticians trained in one methodological paradigm lost their professional standing when a new paradigm revealed the limitations of the old. The resistance is not irrational. It is the sound of genuine expertise being reorganized by a force that does not respect the categories in which that expertise was developed.

The fourth stage is adaptation — the period during which the culture builds the institutional infrastructure that the technology's reliable use requires. This is the stage that Daston's research illuminates most fully, because the infrastructure is her subject — the standards, the practices, the training programs, the evaluative conventions, the moral economies of trust that together constitute the epistemological architecture of a knowledge-producing community. The adaptation is slow because it is institutional rather than individual, contested because it redistributes authority, and incomplete because the technology continues to evolve faster than the institutions can respond. But it is real. The evaluative practices that emerge from the adaptation phase genuinely improve the calibration of trust — they reduce the gap between the technology's implicit claims and its actual reliability, and they provide the scaffolding within which individual users can develop calibrated judgment.

The fifth stage is what might be called settled use — the long period during which the technology is deployed within the institutional framework that the adaptation phase produced, with calibrated trust, recognized limitations, and established practices for managing the gap between what the technology promises and what it delivers. Settled use is never perfectly calibrated. The institutions are never fully adequate. New failure modes emerge as the technology evolves, and the institutional response is always playing catch-up. But settled use is qualitatively different from the uncalibrated use of the exhilaration phase, because the user community has developed the collective knowledge, the shared standards, and the institutional support that enable trust to be extended in proportion to evidence rather than in proportion to the technology's confidence artifacts.

The current moment, evaluated against this pattern, is situated at the late edge of the exhilaration phase and the early edge of the adaptation phase. The resistance is well underway — visible in the debates over AI in education, in professional practice, in creative work, in the labor market. The adaptation has barely begun. Some organizations have developed internal guidelines for AI use. Some professional societies have issued preliminary standards. Some educational institutions have begun to develop curricula for what might be called AI literacy. But these efforts are fragmentary, underfunded, and operating at a pace fundamentally mismatched with the technology's deployment.

Daston's historical framework permits an assessment of what the pattern predicts for the AI transition — with the caveat that historical patterns are not deterministic and that the specific features of the current transition may produce outcomes that deviate from the pattern in ways that cannot be fully anticipated.

The pattern predicts, with high confidence, that the institutional adaptation will arrive later than the technology requires. This prediction is based on the structural mismatch between the speed of technological deployment (which moves at the speed of the market) and the speed of institutional development (which moves at the speed of deliberation, negotiation, and political will). The mismatch has characterized every previous transition, and there is no structural reason to expect the current transition to be different. The institutions will be late.

The pattern predicts, with moderate confidence, that the period of costly errors will be more severe than for previous transitions. This prediction is based on three features of the current transition that have no close historical precedent: the invisibility of AI's errors in its outputs, the power of AI's confidence artifact (which operates in the universal medium of natural language rather than in a domain-specific format), and the speed at which AI is being deployed (which compresses into months what previous transitions distributed across decades). Each of these features makes the calibration challenge harder, and the combination makes it harder still.

The pattern predicts, with lower but still meaningful confidence, that the adaptation will eventually succeed — that evaluative institutions adequate to the technology's characteristic modes of failure will eventually be built, that the gap between AI's implicit claims and its actual reliability will eventually be narrowed by institutional mechanisms, and that the culture will eventually develop the calibrated trust that enables the technology's genuine benefits to be realized without the systematic overconfidence that currently accompanies its use.

But the word "eventually" conceals the variable that matters most: the length and severity of the period between the technology's deployment and the institutional response. That period is where the costly errors accumulate, where capacities are eroded, where the evaluative competencies that calibration requires are most needed and least available. The length of the period is not determined by the technology. It is determined by the institutional response — by the speed, the quality, and the adequacy of the evaluative infrastructure that the culture builds around the technology.

Daston offered a formulation in her work on algorithmic governance that captures the stakes with characteristic precision. She identified three enduring kinds of rules that have operated throughout human history: the algorithms that calculate and measure, the laws that govern, and the models that teach. AI operates primarily in the first register — it calculates and measures, matching patterns across vast datasets with a speed and comprehensiveness that no human can match. But the evaluation of AI's outputs requires all three registers: the algorithmic competence to understand how the technology produces its outputs, the governance structures that hold the technology accountable for the reliability of those outputs, and the pedagogical practices that teach users to assess the outputs with the independence and critical acuity that calibrated trust demands.

The current institutional landscape provides the first register inadequately, the second barely at all, and the third not yet in any systematic form. The gap is not a prediction. It is a description of the present situation, assessed against the institutional requirements that the historical pattern identifies as necessary for the calibrated use of any powerful knowledge-producing technology.

The pattern can be broken. Historical patterns are not natural laws. They are tendencies produced by specific structural conditions, and if the conditions change, the tendency can be altered. The condition that most powerfully determines the length of the costly-error period is awareness — the recognition, by the relevant institutional actors, that the pattern exists and that deliberate effort is required to shorten the gap between technology and institutional response. Previous transitions unfolded without this awareness, because the pattern had not been identified. The present transition has the advantage of historical knowledge — the accumulated evidence of five centuries of technology transitions, documented with enough precision to identify the structural features that produce costly errors and the institutional interventions that reduce them.

Whether this advantage will be exploited — whether the recognition of the pattern will translate into the institutional action that the pattern demands — is the question that the present moment is answering. The answer is not yet determined. The institutions are being designed, debated, funded, and implemented at a pace that reflects the competing pressures of technological momentum, economic incentive, political will, and the inherent difficulty of building evaluative infrastructure for a technology that is still evolving. The outcome depends not on the technology's capabilities but on the quality of the human response to those capabilities — the quality of the institutions, the standards, the educational practices, and the moral commitments that the culture brings to the challenge of learning to live wisely with a powerful new way of producing knowledge.

Daston, in a podcast conversation about the history of rules and algorithms, described AI as "our first encounter with an alien form of intelligence" and argued that "it's really stupid to try and make it the same as our intelligence." The remark cuts deeper than its colloquial register might suggest. The attempt to evaluate AI's outputs using the heuristics developed for evaluating human-produced knowledge is precisely the mistake that the history of objectivity warns against — the mistake of applying the evaluative framework of the previous technology to the outputs of the new one, and finding the framework inadequate only after the costly errors have accumulated.

The alternative is to develop evaluative frameworks specific to the technology's actual characteristics — frameworks that account for the decorrelation of fluency and authority, for the invisibility of errors in fluent prose, for the structural impairment of the calibration mechanism, for the prospective erosion of evaluative capacities. These frameworks will not look like the frameworks developed for photography or statistics or computational modeling, because the technology's characteristics are different. But they will serve the same function: they will provide the institutional scaffolding within which calibrated trust can develop, individual evaluative competencies can be exercised and maintained, and the gap between the technology's claims and its reliability can be narrowed to a tolerable width.

The history does not predict a happy ending. It does not predict a catastrophe. It predicts a difficult, contested, institutionally demanding process of adjustment whose outcome depends on the quality of the human effort invested in it. The pattern is clear. The structural conditions are identifiable. The institutional requirements are specifiable. What remains is the effort — the sustained, collective, institutionally organized effort to build the evaluative infrastructure that a powerful new knowledge-producing technology requires and that no previous generation has built quickly enough.

The question posed by five centuries of evidence is not whether the effort will eventually succeed. It is whether it will succeed in time.

---

Chapter 10: The Alien in the Archive

Lorraine Daston, in a 2022 conversation about artificial intelligence, offered a remark that has the quality of an aphorism that has not yet been fully unpacked: AI is "our first encounter with an alien form of intelligence. And it's really stupid to try and make it the same as our intelligence."

The remark is worth lingering over, because it reframes the entire epistemological challenge this book has been analyzing. The history of objectivity, as Daston has reconstructed it, is a history of technologies that extended human capacities for observation, measurement, and analysis — technologies that were understood, correctly or incorrectly, as amplifications of human perception. The microscope amplified the eye. The photograph amplified the record. The statistical method amplified the capacity for detecting patterns in data. Each technology was understood as doing, with greater precision or greater scope, something that humans already did. The evaluative frameworks developed for each technology were therefore extensions of the evaluative frameworks that had been developed for the human capacities the technology amplified. The standards for evaluating microscopic images were extensions of the standards for evaluating visual observations. The standards for evaluating statistical findings were extensions of the standards for evaluating empirical claims.

Daston's characterization of AI as "alien" intelligence suggests that this approach — extending the evaluative framework of human knowledge production to the assessment of AI-generated knowledge — may be fundamentally inadequate. Not because the framework is wrong, but because it was designed for a different kind of epistemic agent. Human knowledge production is embedded in a web of social relationships, institutional accountabilities, biographical specificities, and moral commitments that together constitute the infrastructure of trust. A human expert's authority derives not only from the content of her claims but from her location in a network of institutional affiliations, professional reputations, social obligations, and personal stakes that provide the context within which her claims can be evaluated. The evaluative framework that has been developed for human knowledge production takes this infrastructure for granted. It assumes that the knowledge producer is a social agent — an entity embedded in relationships of trust and accountability, capable of being held responsible for the reliability of its claims, subject to the norms and sanctions of the communities in which it operates.

AI is not a social agent in this sense. It is not embedded in relationships of trust. It cannot be held responsible in any meaningful sense for the reliability of its outputs. It is not subject to the moral economy of the scientific community or any other knowledge-producing community. It operates outside the web of social, institutional, and moral relationships that the evaluative framework of human knowledge production presupposes. It is, in Daston's precise sense, alien — not because it is hostile or incomprehensible, but because it operates according to principles that are categorically different from the principles that govern human knowledge production, and the evaluative frameworks developed for human knowledge production do not map onto it without distortion.

The implication is that the calibration challenge is even harder than the historical pattern suggests. The pattern predicts that evaluative institutions will eventually be built and that they will eventually produce calibrated trust. But the pattern is based on transitions between technologies that amplified human capacities — technologies whose evaluative frameworks could be developed as extensions of existing frameworks. If AI is not an amplification of human knowledge production but a categorically different mode of knowledge production — if Daston is right that it represents an alien intelligence rather than a more powerful version of human intelligence — then the evaluative frameworks must be built not by extension but from the ground up, addressing the specific characteristics of a knowledge-producing process that has no precedent in the history of human epistemology.

Daston's historical research on data provides one starting point for such a framework. In her genealogy of the concept, she showed that "data" — from the Latin dare, to give — originally referred to the premises of an argument, the starting points that were stipulated rather than discovered. The transformation of data from "things given" to "things found" was a centuries-long process that required the construction of elaborate systems of collection, classification, and standardization. The contemporary concept of data as raw material gathered from the world is the endpoint of this historical process, not its starting point, and the concept carries with it the institutional assumptions of the systems that produced it — assumptions about what is worth collecting, how it should be categorized, in what formats it should be preserved, and by whom it should be curated.

AI training data inherits all of these assumptions. It is not a neutral sample of human knowledge. It is a specific collection, shaped by which institutions digitized their holdings, which languages dominated the internet at the time of collection, which formats were machine-readable, which economic structures determined what was worth preserving in digital form. The training data is, in Daston's genealogical sense, "given" rather than "found" — stipulated by the historically specific systems that produced it rather than discovered in the world. And the AI's outputs inherit the biases of the givens in the same way that the conclusions of a deductive argument inherit the biases of its premises.

This genealogical perspective transforms the understanding of AI's errors. The errors are not random failures of a system that usually gets things right. They are systematic reflections of the historically specific, institutionally mediated, materially constrained character of the data on which the system was trained. When an AI mischaracterizes a philosophical concept — as in the Deleuze episode that recurs throughout Segal's account — the mischaracterization is not a glitch. It is a product of the training data's specific representation of that concept, which reflects the specific texts that happened to be digitized, in the specific formats that happened to be machine-readable, curated by the specific institutions that happened to be involved in the construction of the training corpus.

Understanding AI's errors as genealogically produced — as products of a specific history rather than random failures of a general system — changes the calibration challenge. The question is no longer simply "When is AI reliable?" but "What specific historical, institutional, and material conditions produced the data on which this system was trained, and what systematic biases do those conditions introduce?" The question requires not only technical competence (understanding the model architecture) but historical and institutional literacy (understanding the provenance of the training data) — a combination of competencies that no existing educational program systematically provides.

Daston's concept of the "moral economy" of science offers a second starting point. The moral economy is the system of affect, trust, and obligation that governs a knowledge-producing community — the norms that determine who is believed, what counts as evidence, how disagreements are resolved, and what consequences follow from the violation of the community's standards. Every knowledge-producing community develops a moral economy, and the moral economy is constitutive of the community's capacity for reliable knowledge production. Without norms of honesty, there is no basis for trust. Without accountability for errors, there is no incentive for care. Without shared standards of evidence, there is no basis for adjudicating disputes.

AI-generated knowledge is produced outside any moral economy. The system has no stake in the accuracy of its outputs. It faces no consequences for error. It is not embedded in a community whose norms would constrain its tendency toward confident assertion regardless of epistemic warrant. The absence of a moral economy is not a technical limitation that can be corrected by better alignment procedures. It is a structural feature of the technology's nature as an alien intelligence — an intelligence that operates outside the social, institutional, and moral frameworks within which human knowledge production has always been conducted.

The institutional challenge is therefore not merely to build evaluative mechanisms for AI's outputs. It is to construct something analogous to a moral economy around AI's use — a system of norms, accountabilities, and expectations that provides the infrastructure of trust that the technology itself cannot supply. The norms would govern not the AI (which cannot be governed in the relevant sense) but the humans who deploy, interpret, and rely on AI-generated knowledge. The accountability would fall not on the system but on the practitioners who use its outputs without adequate verification, the institutions that deploy it without adequate oversight, and the policymakers who permit its deployment without adequate regulatory frameworks.

This is what Daston's framework ultimately reveals about the AI transition: it is not primarily a technical challenge. It is an institutional and moral challenge — a challenge of building the human infrastructure that a powerful, alien, socially unembedded knowledge-producing technology requires if its outputs are to be used wisely rather than consumed naively. The infrastructure includes evaluative institutions, educational programs, professional standards, and feedback mechanisms. But it also includes something less tangible and more fundamental: a collective commitment to the principle that knowledge carries obligations — that the production and use of knowledge is a moral activity, embedded in relationships of trust, governed by norms of accountability, and subject to standards that no technology, however powerful, can exempt its users from honoring.

Daston concluded her remarks on AI with a warning that has the quiet force of a historian who has spent a career studying the consequences of misplaced trust: the task of debugging algorithms is being outsourced to consumers, who "are now forced to flag downstream effects when things go wrong." The observation identifies the precise location of the institutional failure. The evaluative burden that should be borne by institutions — by the organizations that develop, deploy, and profit from AI — is being transferred to individual users who are least equipped to bear it. The transfer is not deliberate. It is structural — a consequence of deploying a technology without the evaluative infrastructure its use requires.

Reversing the transfer — building the institutions that bear the evaluative burden, developing the standards that codify the lessons of experience, creating the educational programs that equip users with the competencies that calibrated trust demands — is the work that the historical pattern identifies as necessary and the present moment identifies as urgent. The work is institutional. It is collective. It is sustained. And it is, as Daston's forty years of historical research persistently demonstrate, the work on which everything else depends.

The machines have entered the archive. They are producing knowledge at a scale and speed that no previous technology has approached. The knowledge is useful, powerful, and structurally unreliable in ways that the users cannot yet systematically detect. The institutions that should be mediating between the technology's outputs and the communities that depend on them are, at best, in their earliest stages of construction.

The historian's contribution is not to build those institutions. It is to show, with the accumulated evidence of five centuries, that they are necessary, that they are achievable, and that the cost of delay is measured not in abstractions but in the specific, documentable, human consequences of trusting a powerful technology before the culture has learned to tell the difference between what it knows and what it merely says.

---

Epilogue

The phrase I have not been able to let go of is not even the most famous one from Daston's work. It is not "mechanical objectivity" or "truth-to-nature," though both reshaped how I think about what happened when my team started building with Claude. It is a quieter line, from a 2020 interview, almost an aside: "Behind every thin rule is a thick rule cleaning up after it."

I kept returning to it because it described something I had lived but never named.

In Trivandrum, when my engineers started using Claude Code at full capacity, the thin rules were everywhere. The syntax, the boilerplate, the dependency management — all the mechanical, context-free, execute-without-thinking work that Claude handled with stunning fluency. The thin rules were being automated at a pace that felt like liberation. And it was liberation. I said so in The Orange Pill, and I meant it.

But Daston's line named what I had been circling without quite landing on: somebody still had to clean up after the thin rules. Somebody had to decide whether the code Claude generated actually served the product we were building, whether the architecture would hold under load, whether the feature was worth shipping at all. That "somebody" was exercising thick-rule judgment — contextual, discretionary, dependent on years of accumulated understanding that could not be reduced to an algorithm. And the thick-rule work was harder to see, harder to measure, and harder to defend in a quarterly review than the spectacular productivity gains of the thin-rule automation.

What Daston gave me, across the nine chapters of this book, was a framework for understanding why that thick-rule work is not just valuable but existentially necessary — and why every powerful technology in history has created the illusion that it can be dispensed with.

The confidence artifact idea stopped me cold. Not because the phenomenon was unfamiliar — I had described it myself, in different language, when I wrote about Claude's prose outrunning its thinking. But because Daston's historical depth revealed that the phenomenon was not new. It was not a special property of large language models. It was a structural feature of every knowledge-producing technology that humanity has ever trusted. The scientific illustration's beauty. The photograph's sharpness. The statistical table's decimal places. Each one a surface feature that inspired confidence in excess of what the underlying process warranted. Each one invisible to the users who were most susceptible to it, precisely because the feature activated evaluative habits so deeply trained that they operated below consciousness.

And now prose fluency. The confidence artifact that works in every domain, that mimics the markers of genuine expertise so perfectly that the distinction between real and simulated authority requires exactly the kind of independent judgment that the technology's convenience is slowly eroding.

Daston's historical pattern — overconfidence, costly errors, institutional adaptation, calibrated use — gave me something I did not have when I wrote The Orange Pill: a timeline. Not an optimistic timeline. The institutions are always late. The costly errors always come first. The gap between technology and institutional response always widens before it narrows. But the narrowing does come, and it comes because people build the structures that make it possible.

I am one of those people. I build things. It is what I do, and what I will keep doing. But Daston taught me something about what I am building that I had not fully understood: I am not just building products. I am building within an evaluative vacuum — a period in which the institutions that should be calibrating trust in the technology I use every day do not yet exist in adequate form. And the absence of those institutions means that the evaluative burden falls on me, on my team, on every individual user who must decide, with each interaction, whether the polished output on the screen reflects genuine understanding or a statistical echo of patterns in the training data.

That burden should not rest on individuals alone. Daston's entire body of work demonstrates that calibration is an institutional achievement, not a personal virtue. The microscope was not calibrated by individual microscopists exercising personal discipline. It was calibrated by communities that developed shared standards, shared testing protocols, shared training programs. The same will be true for AI. The dams I called for in The Orange Pill — the institutional structures that redirect the river's force toward habitable ground — are exactly what Daston's history shows are necessary and what the present moment has not yet built.

But they will be built. The pattern says so. The question Daston's work forces me to sit with is not whether, but when — and at what cost in the meantime.

I keep thinking about that phrase. Behind every thin rule is a thick rule cleaning up after it. The AI can write the thin rules faster than any human who has ever lived. The thick rules are still ours.

For now.

-- Edo Segal