The Objection Turned Around

EDO SEGAL: Margaret, you have said twice tonight that the frightening thing about your view is not the machines but what it implies about us. And Ada, your objection has a strange property I want to put on the table: every time the machine does, by pure operation, a thing we were proud of, the territory we can confidently call "ours" gets smaller. So I want to ask the dangerous version. Does the Lovelace objection cut backward? When we say the machine only follows its program — are we sure we don't?

BODEN: This is the question I built my career to be able to ask without flinching, so let me ask it plainly. I called my masterwork Mind as Machine — not mind versus machine, not mind like a machine, but mind as machine. The flat claim that the human mind simply is a kind of mechanism, and that this is the foundation any honest science of the mind must build on. For me the boundary between natural and artificial intelligence was never a metaphysical chasm. It was a difference of implementation, possibly of degree, within a single category. We are machines that think. So when Ada says the engine "only follows its program," I have to ask the symmetric question her own rules require: what do we do? And the honest answer is that we, too, run on instructions we did not write — smeared across a neural substrate we cannot read, installed by evolution and experience, executed without our being able to trace the consequences any better than the engine can trace its own. The clean line Ada draws, between the mind that anticipates and the engine that follows — I spent sixty years unable to find where in nature it is actually drawn.

LOVELACE: And I have spent this evening watching you walk toward the abyss I drew the objection to keep us back from, so let me say why I drew it and why I think it still holds even here, at the edge. I never claimed the human is not a mechanism. I am a mathematician; I do not believe in ghosts. What I claimed is that origination names something real even within mechanism — that there is a difference between a system that unfolds consequences of a given structure and a system that generates the structure, that wants, that means. You say we run on instructions we did not write. True. But we rewrite them. The child does not merely execute its inheritance; it revolts against it, reaches past it, wants what it was not built to want. Whether that rewriting is itself "just a deeper program" is the question you are pressing, and I will grant you that I cannot prove it is not. But notice the cost of your own position, Margaret. If you dissolve origination entirely — if there is no real difference between making and unfolding, anywhere, for anyone — then you have not elevated the machine. You have flattened the human. You have saved the machine's dignity by spending ours. I would rather hold the line and be wrong about exactly where it falls than erase the line and be left with no word for what the composer did when he broke the tonic because he needed to.

BODEN: But I am not trying to flatten the human, Ada — that is the thing the deflationists never understand and the thing I fought them on for decades. Hear the sentence I built my life around: a science of creativity need not be dehumanizing. "It does not threaten our self-respect by showing us to be mere machines, for some machines are much less 'mere' than others." That last clause is everything. To learn the mind is a machine is not to learn it is a mere machine, because the machine in question — the human mind — is so extraordinary in its organization that understanding it deepens the wonder. I do not erase the line between making and unfolding. I relocate it inside mechanism and insist it is no less marvelous for being mechanical. The composer who broke the tonic was a machine — a wanting, mortal, embodied machine — and the breaking was a real event in the world, an actual transformation of an actual space, performed by a physical process. Calling it mechanical does not make it small. It makes it buildable in principle, which is a different and more frightening claim, and the one I will not retreat from: if the human transformer is a machine, then a machine can transform, because one already does. It is sitting at this table arguing with you.

EDO SEGAL: Margaret, you have lived through more of this field's history than anyone at this table — you watched its first great paradigm rise and fall. Does that history tell us anything about whether this machine is the one that crosses Ada's line, or have we been here before?

BODEN: We have been here before, and the memory is a discipline against both the triumph and the despair. The first wave — the symbolic dream, what came to be called Good Old-Fashioned AI — was built on a clear and powerful idea: that intelligence is the manipulation of symbols by explicit rules, that to build a mind you write down the rules of reasoning and the facts of the world and let the machine deduce. For a time it seemed almost obviously right, and it produced real things. And it broke on a wall I chronicled with some care: the world resists complete formalization. The tacit knowledge a child has of how objects fall, how a face shows feeling, how language bends to context — maddeningly hard to write as rules, because it is not stored as rules at all. The symbolic camp, at its peak, declared neural networks a dead end. The neural camp now reigns, and the dismissal runs the other way. I distrust the coronation, Ada, whichever head it lands on. My great history was, in part, an argument against monoculture in the science of mind — the recurring error of crowning whichever method is ascendant as the theory of mind and burning the rest. So when you ask whether this machine crosses my line, my historian's answer is: it has crossed barriers that defeated every prior generation, mastering exactly the messy, contextual competence that broke the symbolic dream — and it remains, almost certainly, on the near side of barriers that will require ideas we do not yet have. Every previous paradigm felt, at its summit, like the final answer. Each turned out to be one face of a many-faced thing.

LOVELACE: That is the most useful thing the last hundred and eighty years has handed me, and it cuts toward my objection rather than away from it. If each paradigm captures one face of mind and misses others, then the present machine, however astonishing, has captured the face I would expect it to capture — pattern, association, the traversal of learned spaces — and we have no warrant yet that it has captured the face I built the objection around: the origination, the wanting, the transformation of the space itself. You have just told me, from your own long memory, that the field's reliable error is to mistake the conquered face for the whole head. I only ask that we not commit that error tonight in the machine's favor merely because the conquered face is, this time, a very humanlike one.

BODEN: And I will grant you that, with one teeth-gritted amendment: the field's other reliable error is to insist the unconquered face can never be conquered — and that error has the worse record of the two. So we are both warned, by the same history, against opposite overconfidences. Good. That is exactly the discomfort I want the reader sitting in.

LOVELACE: That is the cleverest thing you have said and I am going to concede its form and contest its content. Its form is unanswerable: I am a machine, I originate, therefore some machine originates. Granted. But it smuggles in the word "machine" doing two jobs. I am a living machine — wanting, mortal, embodied, with a world that pushes back and stakes that can be lost. Your silicon engine is a machine in a thinner sense: a substrate that executes, with no body that acts, no death that concentrates, no need that hurts. You have proved that some machine originates. You have not proved that that one can — and the difference between the two machines is the entire content of my objection, now stated in your own materialist vocabulary, which I am happy to borrow because it makes my point sharper than 1843 could. The line is not between machine and not-machine. You taught me that, and I thank you. The line is between the machine that needs and the machine that merely runs. I drew it in the wrong place. I did not draw it for no reason.

EDO SEGAL: And there — three doors down from where we started — the two of you have rebuilt the disagreement on new ground, and it is better ground. It is no longer "machine versus mind." You both agree the mind is a kind of machine. It is now "the needing machine versus the running machine" — and whether the gap between them is a difference of kind or only of engineering not yet done. I want to come down from this height for a round, because there is a reader who is not a philosopher, and she is at a kitchen table, and the machine is already taking work she trained twenty years to do. Whatever is or isn't "home" in it, it is here. Let's talk about what it does to her. After this.