The Lovelace Question

EDO SEGAL: There's a ghost at this table and I want to name her. Ada Lovelace, 1843, the first program ever written and already the first deflation: the Analytical Engine, she said, has no pretensions to originate anything — it can do only whatever we know how to order it to perform. The Lovelace objection is the great-grandmother of the stochastic parrot. Turing took her seriously enough to answer her by name a century later, and his answer — that machines take us by surprise constantly — Margaret, you've called both right and beside the point. Untangle it. Because I think the public's entire confusion about machine creativity is a confusion about surprise.

BODEN: Let me first pay the lady the respect of quoting her exactly, because the misquotations have done a century and a half of damage in both directions. Note G of her commentary on Menabrea, 1843: "The Analytical Engine has no pretensions whatever to originate anything. It can do whatever we know how to order it to perform." Two sentences, and observe — as Turing observed — that the second is a conditional that the first dresses as a verdict. Whatever we know how to order it to perform — but what if we know how to order it to perform learning? To order it to acquire orders we never gave? Ada, who understood the engine better than anyone alive including Babbage, could not have anticipated that the ordering itself would become generative — though I have always suspected, from the rest of Note G, which is visionary about the engine composing music if the science of harmony were suitably expressed, that she would have relished the loophole rather than resented it. She is claimed by both camps, Edo, and belongs to neither: she deflated the engine of her century with perfect accuracy and left the door open for ours with perfect precision. The untangling of what came after requires my favorite distinction of the evening. Surprise comes in kinds, and the kinds correspond to my doors. There is the surprise of the improbable — the unexpected combination, the statistic outlier; one gasps, briefly. There is the surprise of the unvisited — I never realized the style allowed that; one's map of a known country grows. And there is the third surprise, the transformational one, which does not feel like either of those. It feels, first, like impossibility — like a category error, like nonsense — because the object violates the very rules by which you parse the domain. Atonal music was not heard as new music; it was heard as not music. The ring was not a new chain; it was the end of chain-thinking. That third surprise has a signature: initial rejection, by experts especially, followed — sometimes decades later — by the reorganization of the whole field around the impossible object. Turing was right that machines surprise us; Lovelace was right that the engine originates nothing; and both were beside the point because neither asked which kind. The evening's empirical question, stated at last in its sharpest form: has any machine yet produced surprise of the third kind? And there I must be the scientist rather than the advocate: I have not seen it. The alien move shocked the grandmasters — and they assimilated it within months. That is surprise of the second kind, the greatest second-kind surprise in history, perhaps. The third kind remains, to date, a human monopoly.

EDO SEGAL: Stay with the lottery of timing for a second, Margaret, because you said earlier that H-creativity is P-creativity that wins a lottery, and I want the listener to feel how literal that is.

BODEN: Brutally literal, and the patron saint is Gregor Mendel. The man discovered the laws of inheritance — discovered them, published them, correctly — and the field declined to notice for thirty-four years. The P-creative act was complete in 1866; the H-creative verdict arrived in 1900, when three other botanists reached the same place independently and only then found the monk's paper waiting for them like a note from the past. Now ask: what was Mendel's creativity made of during the thirty-four years? It did not increase in 1900. Nothing happened in Mendel — he was dead. What happened was social: the community became capable of receiving the idea. This is why I refuse to let the value question swallow the mechanism question. If creativity were constituted by communal uptake, Mendel was not creative until sixteen years after his funeral — which is a sentence only a sociologist could love. The mechanism was in the man; the lottery was in the calendar. And the relevance tonight is exact: when Emily says the machine's candidates are crowned by humans, she is describing the lottery. The question on the table has always been whether there is a Mendel in the machine.

BENDER: Which is a remarkable framing to have to answer — and I'll answer it by holding the room still around what Margaret just conceded on her way past it: the founder of the computational theory of creativity, reporting that the crown jewel remains unexhibited. Now let me say why I think the monopoly is not an accident of engineering schedule. Margaret's third surprise has that social signature — rejection, then reorganization — because transformation is a negotiation. The transformer breaks a rule and is answerable for the breakage: Schoenberg had to live inside the scandal, argue, teach, compose through the rejection, stake a life on the claim that the broken rule was a door and not a vandalism. The answerability is not decoration around the creative act. It is the mechanism by which a community distinguishes transformation from noise — and answerability requires someone who can be answerable. A system that cannot be wrong in a way that costs it anything cannot make the kind of claim that transformation is. It can emit the artifact. It cannot stand behind it. That's why the third kind hasn't appeared, Edo, and why I predict it won't: not because the generator is weak, but because emission isn't assertion. The bottleneck isn't in the candidate-space. It's in the standing.

EDO SEGAL: Margaret, before your reply — Turing deserves his moment in this chapter, since he's the one who answered Ada by name. Nineteen fifty, the Mind paper, the section he called Lady Lovelace's Objection. His counter was the surprise argument — machines take me by surprise with great frequency — and you've said it was both right and beside the point. But he had a second counter people forget: that the objection assumes humans are never themselves doing merely what they were ordered — that originality in us might equally be the working-out of orders we received from education, experience, inheritance. He turned Ada's deflation around until it pointed at everyone. Was that move legitimate?

BODEN: Legitimate, profound, and double-edged in a way Turing — who enjoyed edges — fully intended. If "it only does what it was ordered to do" defeats machine originality, then a sufficiently complete neuroscience defeats yours and mine: we, too, unfold from specifications we did not write. The objection, taken seriously, abolishes creativity everywhere, which is a reductio — creativity exists; I have met it. So the deflation must be wrong somewhere, and locating the wrongness is precisely my life's work: it is wrong because ordering is the wrong level of description. A system of sufficient richness, given orders, can enter regions of behavior the order-givers could not have anticipated even in principle — and at that point "ordered" has become a genealogical fact about the system, not a bound on its conduct. The interesting boundary was never ordered-versus-free. It is anticipatable-versus-not, and that boundary runs through the middle of the machine question rather than neatly around it. Which brings me to the reply I owe Emily — unless, and I offer this with the discomfort it costs me, the standing can be borrowed. Schoenberg's answerability was his own. But consider the system embedded in a human practice — a laboratory, say, where the machine proposes the impossible molecule and the humans stake their careers on synthesizing it. The claim is made by the partnership. The candidate came from the machine; the standing came from the people; the transformation, if it lands, belongs to... whom? Emily's framework wants a single owner of the act. The laboratories of the next decade, I rather suspect, will not supply one.

EDO SEGAL: Borrowed standing. That phrase is going to follow me around — so let me test it where it lives already, because the laboratories Margaret describes are not hypothetical. The protein-structure systems propose folds; crystallographers stake beam time on them. The mathematics systems propose conjectures; humans stake careers on proofs. There's a generation of scientists forming right now whose daily epistemic posture is the machine suggested it, we verified it — and Margaret, the speed is doing something to the sociology you should both weigh in on. When verification lags proposal by months, the proposing system accumulates a kind of de facto authority — papers cite its outputs as if they were findings, journalists report candidates as discoveries. The borrowed standing starts flowing backward up the pipe: the instrument gets the headline; the verifiers become, in the public account, technicians of the machine's insight. Who is answerable in that laboratory, five years out?

BODEN: The honest answer is: the answerability is being diluted, and dilution of answerability is how fields rot — we knew that before any of this; ask the replication crisis. My partnership model requires that the humans' stake stays load-bearing — that someone's name, career, and nerve ride on each crowned candidate. If the sociology drifts to where the machine's fluency pre-crowns its own candidates and humans merely rubber-stamp, then Emily's nightmare and mine converge: form would be performing the social functions of meaning, at scale, inside science itself. I said the machine is on the team. I never said it should captain it, and a team whose captain cannot be embarrassed has no captain.

BENDER: Mark that we agree completely here, because it's the most practically urgent agreement of the night: fluency must never pre-crown. Every institutional design question of the decade — peer review, court filings, medical records, journalism — reduces to keeping a human's revocable, embarrassable signature between generated candidates and consequential acts. Now, Edo — you were working up the nerve to make it personal.

EDO SEGAL: I was. When my book went to print with the machine's sentences in it under my name — was that borrowed standing? I answered for every sentence. I would have eaten the failure of any of them. Emily, by your own logic — didn't my answerability make those sentences meant?

BENDER: Yes. Yours. That's been my point all evening, Edo. The meaning arrived exactly when you signed.

BODEN: And the candidates arrived, Edo, before you signed — from somewhere. That has been mine.