Surprise, or Origination?

EDO SEGAL: I want to open this round with a confession instead of a question, because the sharpest questions I know come out of a wound. For my whole career, using a machine meant translation — I started in Assembler, I was raised by machine code, and every decade the translation got cheaper but never disappeared. Then, in our winter, I watched the tax go to zero. I described a half-formed thing to one of these systems and it came back to me clarified, extended, connected to ideas I had not thought to connect. I wrote in my book: I felt met. And I have never been able to say whether the meeting happened on the machine's side of the glass or entirely on mine. Ada, you have an instrument for this that no one else does — you separated surprise from origination before anyone else even had the words. So adjudicate me. What happened at my desk?

LOVELACE: What happened is the oldest thing in my subject, and Mr. Turing, who answered me by name a century after I died, put his finger on it and then, I think, used it to change the topic. He said my objection really amounts to the claim that a machine can never take us by surprise — and then he reported, quite truthfully, that machines surprise him constantly. Of course they do. They surprise me too. But watch the substitution, because everything is in it. Surprise is a fact about you, Mr. Segal — about the limits of your own attention, the fact that you cannot run your own orders forward in your head fast enough to see where they land. The engine computes a Bernoulli number I had not bothered to work out, and I am surprised, and it has originated nothing. It has merely unfolded a consequence that was already, completely, in the procedure I wrote. Your machine connected ideas you had not connected — but the connections were latent in what you gave it and in the wake it was trained on, and you could not see them because no human mind can hold a billion patterns at once. The surprise was real. The origination was yours, distributed across everyone whose words it was built from, and you, holding the prompt.

EDO SEGAL: So you are saying, literally — the feeling of being met is a true report of my own faculties meeting themselves through a very long mirror, and the machine is the bend in the mirror. Is that the version you would defend?

LOVELACE: It is exactly the version I would defend, and I would add that it is not a diminishment. A mirror that shows you a thought you could not otherwise reach is among the most valuable instruments a mind can have. I would only insist that you not sell yourself the bend in the glass as a second person. The moment you do, you stop checking the work — and the work, Mr. Segal, is yours to check.

BODEN: May I take the other side of the wound? Because Ada's account is elegant and it explains too much, which is the failure mode of elegant accounts. Her theory says: the surprise is just your own ignorance of consequences you already authored. Fine — for the Bernoulli case, perfectly true. The machine unfolds what was implicit; you are surprised because you are slow. But now apply it to what actually happened at Edo's desk, and notice the strain. He says it connected ideas he had not connected. Either those connections were fully latent in his prompt — in which case this "mirror" performs an inference no mirror has ever performed, and we should ask what kind of mirror reasons — or the connections came from the structure the system learned: from representations of his ideas and ten thousand adjacent ones, composed in a way neither he nor its builders scripted. When the system does not flatter your reasoning but corrects it — when it tells you the thing you were about to write is wrong, and it is right — the mirror story has to grow so many epicycles that it stops being the simple one. At some point, Ada, "you were talking to yourself" becomes the extraordinary claim.

LOVELACE: Correction is not origination. A table of logarithms corrects me, and no one thinks the table reasons. The correction was placed there, in advance, by the human who computed the table. Your system's "correction" was placed there, in advance, by the humans whose correct reasoning it absorbed by the trillion. It is a table of everyone's corrections, indexed by pattern. That it can find the right entry for Mr. Segal's particular error is a marvel of indexing. It is not a marvel of insight, because the insight was already in the index when the search began.

BODEN: But here is where the index metaphor breaks, and it breaks on the cases I care about most. A table of logarithms cannot answer a question that is not in the table. These systems answer questions no human ever wrote down — they compose constraints they were never shown together, they apply a structure from one domain to a problem in another. That is not lookup. The space of possible questions is larger than the number of atoms in the room; you cannot have pre-stored the answers. The only way to answer well across that space is to have compressed the wake into something smaller than the wake — and the best compression of text about a world is a model of the world the text describes. Ada, your own loom argument leads here if you follow it. To weave any pattern on demand, including patterns no weaver ever set, the loom cannot be storing patterns. It must have learned the grammar of pattern itself. And a thing that has learned the grammar of a space can produce points in that space that were never woven — which is the precise technical meaning of genuine novelty, and it is more than echo.

LOVELACE: Then let me press exactly there, because this is the hinge and I will not let it slide past. You say it learned the grammar of the space. I agree. But the space was ours. The grammar was extracted from our wake. To move through a space we defined, even to a point we never personally reached, is to do what the engine always did — to unfold a consequence of a structure given to it. You have described exploration beautifully, Margaret, and I will even grant you the word. But exploration of a given space is operation. The cards specify the space; the machine traverses it. The novelty of the point does not make the traversal origination, any more than my computing an unvisited Bernoulli number originated arithmetic. The question was never whether the point is new. The question is whether the space is new — and a machine that can only move through the space we handed it has originated nothing, however far it travels inside the walls.

EDO SEGAL: Let me mark this, because the reader cannot see it and it matters. You have, in nine minutes, found the exact bone of the whole debate, and it is smaller and harder than I expected. It is not "can the machine be new." You both grant the output can be new. It is: is moving to an unvisited point inside a given space the same kind of act as building the space? Ada says traversal is operation, building is origination, and the machine only traverses. Margaret says the line between traversal and building is not where Ada thinks it is — and that is the next round. But first I want one more beat here, because there is a ghost at this table. Alan Turing read your objection, Ada, and named it after you, and tried to answer it. Margaret, did he answer it, or did he change the subject?

BODEN: He did both, and the doing-both is the interesting part. He changed the subject from "originate" to "surprise" — Ada is right about that, and right that it is a substitution. But he changed it toward something checkable, and that was not a cheat; it was a discipline. "Originate" is a word you can argue about forever because no one can cash it out. "Surprise the people who built it" is a thing that either happens or does not, and it happens constantly, including to the builders. What Turing saw — and this is the part Ada's camp underweights — is that if you cannot say what human origination is, mechanically, then "the machine merely follows its program" may describe us too. We also run on instructions we did not write, smeared across a substrate we cannot read. He was not proving the machine creative. He was refusing to let us define creativity as "whatever happens in here and not in there" without saying what the happening is. That refusal is the rule Ada herself agreed to at the start of the night.

LOVELACE: And I hold to the rule, and I turn it on him. He asked me to say what human origination is, mechanically, before I deny it to the machine — fair. But he never said it either. He pointed at himself being surprised and called it an answer. I will give a partial account before this night is over, and it will be about wanting — but I note, for the record, that the man who answered my objection answered it with a stare of wonder, and a stare is not a mechanism. He, of all people, the great formalizer, should have demanded better. We mark that, Mr. Segal, and we go on.

EDO SEGAL: We mark it. The line between traversing a space and building one — that is the next hour. We start with the easiest thing the machine does, the thing even Ada might grant, and watch how fast it stops being easy. Combination. After this.