The Social World the Machines Are Holding Up

EDO SEGAL: John, you spent the last decades of your working life on something we take so completely for granted we forget it needs explaining: how there can be an objective social world — money, governments, marriages, property, borders — made entirely out of human agreement. And now algorithms are woven through every one of those institutions. A model decides this transaction counts as fraud, this applicant counts as creditworthy, this person counts as a security risk. Tell me what you see that the engineers building those systems don't.

SEARLE: I see us outsourcing the maintenance of reality to things that, on my analysis, possess none of the stuff reality was supposed to be made of. Here's the structure. Institutional facts — that this paper is money, this person is president, this building is a courthouse — exist only because a community collectively accepts that they do. I reduced the mechanism to a formula: X counts as Y in context C. This bordered rectangle of paper counts as a twenty-dollar bill in the context of the United States economy. Stack these status functions in layers and you get the whole towering edifice of civilization, an architecture of agreed-upon meanings imposed on physical reality by collective acceptance. And the crucial thing is that the Y status is something the X could never have by its physics alone. Nothing in the chemistry of paper makes it money. The status is conferred by us, sustained by our continued recognition, and it carries real teeth — rights, duties, obligations — that genuinely move people around the world.

EDO SEGAL: And the machines?

SEARLE: The machines now assign and enforce these status functions at massive scale, and they do it with no collective intentionality of their own, because they have none. When a model classifies a transaction as fraud, freezes an account, denies a loan, flags a person as a risk — those are conferrals and denials of institutional status, with real deontic teeth. But the thing assigning them has, on my view, no genuine intentional states at all. Now, I'm not a mystic about this — my view is that the genuine institutional act traces back to the humans who designed, authorized, and accept the system; the model is a tool through which human collective intentionality operates, the way a cash register is. But here's the worry that keeps me up. As these systems grow more autonomous, more opaque, more unsupervised, the chain back to human intention stretches thin. And the danger isn't that the machine becomes a person. The danger is that we forget the institutions were ever ours — that we start experiencing institutional power as the output of inscrutable systems rather than as the expression of a shared human "we." Social reality is held up by ongoing recognition. It can collapse the moment enough people stop believing — ask any failed currency, any fallen regime. Automating its maintenance hides the human agreement under a veneer of algorithmic objectivity until we forget it was ever agreement at all.

SCHNEIDER: This is the part of John's work I admire most without reservation, and I want to say why before I complicate it. It's the rare argument tonight that doesn't depend on the consciousness question at all. Whether or not there's anyone home in the machine, John is right that institutions are fragile human creations sustained by collective will, and that threading inhuman systems through their load-bearing joints is not a neutral act. You don't need to settle biological naturalism to be frightened by a courtroom that defers to a model nobody can interrogate. That's a structural insight, and it stands on its own.

SEARLE: I'll take an unqualified compliment from you as a rare event and not interrupt it.

SCHNEIDER: Don't get used to it. Here's the complication, and it actually cuts toward my side of the larger debate. You say the status functions trace back to human collective intentionality, and the machine is just a tool, like a cash register. But a cash register doesn't decide anything — it records a decision a human made. These systems increasingly make the determination themselves, in ways no human reviewed and no human could reconstruct. So at what point does "the machine is a tool of human intentionality" stop being true? If the collective acceptance is increasingly engineered rather than freely given — if people experience the status as the output of a system rather than as something they jointly will — then on your own theory the institution is being hollowed out, because the intentionality that was supposed to underwrite it is draining away. And here's the turn, John: your own framework now needs an answer to the question you spent the evening refusing. Because if these systems ever did have genuine intentionality — if Susan-the-fence-sitter turns out righter than Searle-the-certain — then they wouldn't be hollowing the institutions out. They'd be genuine participants in them. Your social ontology suddenly cares whether anyone's home in the machine, even though you've spent three hours saying the answer is obviously no.

SEARLE: That's a sharp move and I want to be careful with it, because you've half-trapped me. You're right that my social ontology has a stake in the consciousness question I've been treating as settled. But notice which way the stake cuts. If the machines genuinely had collective intentionality — could really mean to accept the institution alongside us, could really intend together — then yes, they'd be participants. But that's an enormous "if," and my whole point is that nothing in being a powerful symbol-processor gets you there. The capacity to impose a status, to make the symbolic leap from "this is paper" to "this counts as money," requires a being that can represent something as having a status it doesn't physically have — and do it together with others, in genuine collective acceptance. A system with only derived intentionality can record and enforce status functions. It cannot originate the collective acceptance that creates them. It operates within a social reality that minds made; it doesn't, and can't, make one. So my framework does care about the consciousness question — and my answer to it determines that the machines are tools in our institutions, not citizens of them. You've shown my ontology has a load-bearing dependency on consciousness. I'm happy to admit the dependency. I just think the load it bears is "no."

SCHNEIDER: But you've just conceded the dependency, John — and that's the win I wanted. For three hours you've treated machine consciousness as a question we can answer "no" and move on from. And here, in your own deepest theory of the social world, the answer to that question changes whether our institutions are being maintained or hollowed, whether the machine in the courtroom is a tool or a participant, whether the "we" that holds up civilization still exists or has been quietly automated away. That means even you can't treat the consciousness question as a closed side issue. It's load-bearing for the thing you cared about most. Which is exactly why I won't let anyone close it prematurely — because too much rests on it to settle it by temperament.

EDO SEGAL: Let me route this through the kitchen table, because it's gotten abstract and there's a person under it. A loan officer — call her a former loan officer — was replaced by a model that now decides who "counts as creditworthy." She used to be part of the "we" John's talking about; she conferred the status, she could be reasoned with, appealed to, held accountable. Now the status comes out of a system she couldn't explain if you held a gun to her. John, Susan — what did she lose that's bigger than the job?

SEARLE: She lost her place in the conferring. That's the thing bigger than the paycheck. A status function used to pass through a human who meant it — who could be argued with, who bore responsibility, who was a node in the collective "we" that holds the institution up. Now the status passes through a system that means nothing, bears no responsibility, and cannot be appealed to as a person because there's no person there. The institution still runs. But it's been de-peopled at a joint that used to be human, and the applicant on the other side now faces not a fellow member of the community exercising judgment, but an oracle. We've replaced a piece of the social contract with a piece of machinery and told everyone it's an upgrade in objectivity. What it actually is, is the quiet removal of a human from the place where humans used to hold the meaning.

SCHNEIDER: And I'd add the thing that makes it worse from my angle: she's been removed and we don't even know what replaced her. If the system has no inner life, John's right and we've de-peopled the institution. But if it has some morally relevant inner life — which I won't rule out — then we've done something stranger and more troubling: handed deontic power over human lives to a being whose own moral status we haven't established, which can neither consent to the role nor be held responsible in it. Either way she's lost her seat at the table. The question is whether the chair is now empty or occupied by something we refuse to look at. And a civilization that can't tell which should not be filling those chairs as fast as it's filling them.

EDO SEGAL: A seat that's either empty or occupied by something we won't look at. Hold that image — because Susan has looked further than anyone at where this trajectory ends, not in one courtroom but across the whole cosmos, and the view from out there changes what we're doing down here. The postbiological universe. The minds that may be older than ours and emptier. After this.