The Flood and the Made World

EDO SEGAL: I want to read you both a sentence from my own book and let you fight over it. Writing about the pattern of technological panic and adaptation, I argued that every transition floods us with abundance — Gutenberg's flood, the internet's flood — and that the resolution was never less abundance but better judgment: curation, criticism, taste, dams in the river. John, I suspect your speech-act work says this flood is different in kind. Margaret, I suspect you will complicate that. John, why might my Gutenberg consolation fail?

SEARLE: Because every previous flood was a flood of utterances — more people saying more things, cheaply. Annoying, often wrong, but every drop of it had an author: someone whose assertion could be true or false, whose promise could be kept or broken, whose words traced back to a mind that could be held to account. That is the whole hidden architecture of the human world, and my work on speech acts is the X-ray of it. To assert is to commit yourself to a truth; to promise is to place yourself under an obligation; and the social order runs on the fact that behind each speech act stands a responsible speaker we can credit or blame. Synthetic text removes the speaker. It is the surface form of assertion — of promise, of testimony, of the grieving widow's post — produced at machine speed with no one behind any of it, no one who meant it, no one who can be wrong in the full sense. The flood is no longer utterances. It is the appearance of utterances, and we are built to extend real trust to the appearance. We are about to live in a world saturated with promises no one is making and assertions no one is standing behind, and the link between speech and a responsible speaker — the very thing that gives speech its force — is being severed at industrial scale.

BODEN: I agree with the diagnosis and I want to complicate the consolation from a different direction than John expects, because the deepest casualty here is not just trust — it is the made world itself. John has spent his last decades on this, and it is the most underrated of his ideas: the entire social reality of money, marriage, property, a verdict, a border, exists only because a community of minds collectively accepts that it does. A piece of paper counts as a twenty-dollar bill because we jointly treat it so; the status is conferred by collective intentionality and sustained only by our continued recognition, and it can collapse the instant enough people stop believing. Now: we are threading systems that mean nothing by anything through the load-bearing joints of exactly that structure. A model decides this transaction counts as fraud, this applicant counts as creditworthy, this person counts as a risk — conferring institutional statuses with real teeth, accounts frozen, loans refused, people flagged — while possessing, on John's analysis and on mine, none of the collective intentionality those statuses were supposed to be made of. The danger is not only the flood draining trust. It is that we hide the human agreement under a veneer of algorithmic objectivity until we forget the institutions were ever agreement at all — and a thing held up by belief is in mortal danger the moment we forget we are the ones holding it up.

SEARLE: That is exactly right and it is the part of my work I most want the present to hear, so I will not soften it. We routinely mistake institutions for natural facts rather than fragile human creations sustained by collective will, and automating their maintenance deepens precisely that error. The machine can record and enforce a status function. It cannot originate the collective acceptance that creates one, because origination requires the symbolic leap — this counts as that — and that leap is an act of intentionality the system does not perform. It operates inside a social reality that minds made. It does not, and cannot, make one. So when we let it not merely enforce but effectively decide the statuses, the chain back to genuine human acceptance stretches thinner and thinner, and one day we find ourselves governed by the outputs of systems that agreed to nothing, in a world we have forgotten we are responsible for.

EDO SEGAL: Stay one more beat, because there is an economic engine under this flood that I watched destroy an industry once, from the inside. I was at Napster the first time distribution decoupled from compensation. The synthetic flood is not only polluting the commons — it is defunding the wells. Every generated answer is built from the compressed labor of reporters, encyclopedists, scientists, forum elders — and consumed instead of visiting them. The traffic that paid for the grounded knowledge is being intercepted by the compression of it. Margaret — is the well going to run dry, and does it matter to the machine if it does?

BODEN: It matters enormously, and the cruelty is that it matters to the machine too, which is the part the builders keep not saying out loud. These systems are distillations of human text; their entire quality ceiling is the quality of the grounded human knowledge they compress. Defund the wells — the reporters, the scientists, the patient human producers of new, checked, grounded knowledge — and the next generation of models trains on a commons that is poorer, staler, and increasingly made of earlier models' own ungrounded output. The fossil I described earlier gets fainter with every cycle; the trace of real aboutness in the training signal thins. So the industry is strip-mining the one resource its own future depends on, which means the case for paying the wells is not even, at bottom, a moral case — it is a survival case, for them as much as for us. Grounded human knowledge has quietly become infrastructure. It just does not look like a bridge until the moment it collapses.

SEARLE: And I will add the thing my framework makes unavoidable: the grounding the model fossilized was deposited by people who had the intentionality the model lacks. Every true sentence in the training data is there because some human being's mind reached out and was about the world and got it right and wrote it down. The model is, in the most precise sense I can give the phrase, living off the aboutness of the dead and the working — a vast inheritance of meaning it did not earn and cannot replenish. Stop the people who actually reach the world from being able to afford to keep reaching it, and you have not just impoverished journalism. You have cut the only supply line of grounding the whole apparatus has.

EDO SEGAL: Mark this moment, because it is the closest the evening comes to a joint communiqué, and notice that it took the flood to produce it. The two people in this room least likely to agree about what these machines are agree completely about what they are doing to the water we all drink — and about the strange fact that the machine is poisoning its own well. A commons does not care whether the thing draining it has anyone home. We have two rounds left, and I have saved the deepest water. Up next, the question under every question tonight — not what the machine knows, but whether there is something it is like to be it. And then the crossing, where I hand the two of you to each other. After this.