Opening Positions

LOVELACE: Thank you. I want to begin with the machine itself, because almost every confusion in this subject begins with forgetting what the machine actually is, and I have the small advantage of having seen one before there was any fluency to dazzle me. The Analytical Engine was brass and cards. It had a store and a mill — what you would call memory and processor — and it was directed by punched cards borrowed from the silk-weaving loom. And here is the thing I understood that even Mr. Babbage, who built it in his head down to the last wheel, did not fully grasp: the engine's behavior was not in its wheels. It was in the cards. The same engine, fed different cards, would do entirely different work. The machine was its program. The wheels were a stage; the program was the play.

Hold that, because everything follows from it. If the behavior lives in the program, and the program is authored — written, in advance, by a mind that knows what each instruction does — then the engine can contain no more than its author put into it. It can elaborate the consequences of what it was given, to any depth, faster than any human could; it can surprise you, certainly, because you cannot trace the consequences of your own orders quickly enough to anticipate them. But it cannot reach past them. "It can do whatever we know how to order it to perform" — whatever we know how to order. The reach of the machine is exactly the reach of what we already, in principle, know how to specify. It has, I wrote, no power of anticipating any analytical relations or truths. It weaves the pattern; it does not mean the pattern, and it does not invent the loom.

Now I am told that the engine has changed — that your modern machines are not fed cards written by a knowing hand but are trained, that their billion instructions are found by a search rather than authored by a person, and that no one can read them. And I have thought hard about this, because it is the only part of the last hundred and eighty years that genuinely tests my objection. But consider what it actually changes and what it does not. The instructions are no longer written by a person — granted. But they are still determined — by the data, which is our wake; by the procedure that searched it, which is our design; by the standard it was trained to satisfy, which is our preference. No one wrote the weights, but everyone wrote the river they were drawn from. The author has not disappeared. The author has been distributed across all of us and then made invisible, which is a very different thing from the author being absent. When the machine surprises you, it has unfolded a consequence of orders so vast and so diffuse that you have lost the thread back to the hand that gave them. That is not origination. That is the loss of an audit trail, mistaken for the birth of a mind.

I will grant the wonder. I granted it in 1843. I said the engine might one day compose elaborate music of any degree of complexity, and I meant it, and I am told it now does. But I said that and I said it originates nothing, in the same Notes, in the same breath, without contradiction — because composing by rule is operation, and operation is exactly what the engine does. To produce the pattern is not to originate it. The machine can give you the music. It cannot want the music, mean the music, or decide that this music and not another was worth making. That deciding — the origination of the point — never entered the cards, and no quantity of cards adds up to it. So when you ask, Mr. Segal, whether the surprise is a new mind: my answer is that it is your own order, returned to you down a channel too long and too dark to see along. You are alone in the room with a magnificent loom. That is my opening.

EDO SEGAL: Margaret.

BODEN: That was the best statement of that position anyone has ever given, which is unsurprising, since she invented it. And I disagree with it less than you might expect and more deeply than is comfortable, so let me be precise about where.

I want to start by giving Ada her due, because the field has spent decades misquoting her. She did not say the engine was stupid, or feeble, or incapable of surprise. She said it originates nothing — a careful, specific boundary. My entire career has been an argument that the boundary is in a different place than she drew it, and that to find the right place you have to stop using the word "creativity" as though it named one thing. It does not. There are at least three different things hiding inside that word, and almost every argument about machine creativity is two people pointing at different ones and shouting.

The first is combinational creativity: making something new by joining old ideas in an unfamiliar way. A horse and a horn make a unicorn. Metaphor works this way, and most of poetry, and the analogies that drive science. The second is exploratory creativity, and here I have to give you a concept I think is the most useful thing I ever made: the conceptual space. A conceptual space is a structured domain of possibilities defined by a set of rules — tonal harmony is one, the sonnet is one, the rules of chess, the conventions of Gothic architecture. Exploratory creativity is moving through that space and finding possibilities that were always permitted by its rules but had never actually been realized. The jazz musician working out the implications of a chord, the mathematician finding a theorem inside an existing axiom system — explorers, all of them. And it is most of what we call creative work.

Now, Ada says the machine produces patterns without originating them — that it weaves without meaning. Watch what my framework does to that. A large language model has learned, from its training, an enormous and intricate conceptual space: the implicit space of plausible text, or images, or music. When it generates, it is moving through that space and returning points within it — and because the space is so vast, it routinely finds regions no individual human has ever personally visited. That is not nothing, Ada. That is exploratory creativity, performed at a breadth and speed no human can match. When it surprises you, sometimes — not always, but sometimes — it has genuinely discovered a possibility that was real, and reachable, and that no one had reached. The surprise is not only about your ignorance of your own orders. It is about the space being larger than any mind that mapped it.

But I have to hold the other half, or I am not being honest, and Ada will catch me if I am not. There is a third kind, the rarest and the highest, and it is the one the machines find hardest. Transformational creativity does not explore a space — it changes the space. It drops or alters one of the defining rules and makes thinkable what was previously unthinkable. When a composer abandons the requirement that music resolve to a tonic, a whole continent of music opens that was strictly impossible under the old rules. A style of thinking, like a road system, can render certain thoughts impossible — unthinkable — and transformation is the rebuilding of the roads. The machines explore brilliantly. Whether they can transform — change the game rather than play it superhumanly well — is, I think, the genuinely open question, and it is open precisely because Ada made it askable. So my opening is this: she is right that much of what looks like creation is operation. And she is wrong that all of it is. The work of the evening is to find the line — and I warn you both, the line moves.

EDO SEGAL: Before I frame the rounds, one discipline I impose on every long conversation, because it pays off two hours later. Each of you, briefly: what do you envy in the other's position? Not respect. Envy. The thing her side gets to have that yours does not. Ada first.

LOVELACE: I envy her instruments. Margaret has tools I did not have — three kinds of creativity, the conceptual space, the distinction between new-to-you and new-to-history. I had a sentence and a table of operations and a great deal of intuition I could not fully discharge. She can take my objection apart joint by joint and say here is the bone that holds and here is the one that breaks, and I had to assert the whole skeleton at once and hope. I gave the field a proposition. She gave it a method. On a bad day I think the proposition was a guess that happened to be deep, and the method is the real science. I envy the method.

BODEN: And I envy her position its floor. Ada gets to stand on something solid — the engine does only what it is ordered, the responsibility runs to the human, the accountability is clean — and push. My position has no floor. I am committed to following the mechanism wherever it goes, and it keeps going to places that dissolve the ground I am standing on. Every time I grant the machine a kind of creativity, I am one step closer to having no principled reason to deny it the next kind, and the next, until I am out past the edge of human specialness with nothing under my feet but "we do not yet know." Ada gets to defend something. I am stuck describing something, and the description keeps getting less flattering to us. People think the frightening thing about my view is the machines. The frightening thing is what it implies about what we were doing all along.

EDO SEGAL: Two openings and two envies, and you can already see the architecture of the night. It is not that one of you loves the machine and one fears it. It is that one of you draws a clean line around the human and the other keeps watching the line move and refuses to lie about it. We start the rounds at the exact seam — at the word your whole disagreement turns on. What, precisely, does the machine do when it surprises us: does it originate, or does it only unfold what we ordered? After this.