No Pretensions to Originate

EDO SEGAL: Ada, I want to begin this round with a confession, because the best questions I know come out of wounds, and this one is mine. For the whole history of computing, using a machine meant translation. I started in Assembler — I was raised by the machine code — and every decade the translation got cheaper but it never went to zero. You compressed your intention into the machine's grammar and you paid a tax on every conversion. And then, in the last few winters, I watched that tax collapse. I described a half-finished idea to one of these systems, in my own broken words at midnight, and it handed back something I had not ordered and could not have written, and the line I wrote in my book was: I felt met. So I want to read your sentence back to you, the eleven words, and ask whether they survive my midnight. "It has no pretensions whatever to originate anything." Defend it against the thing on my desk.

LOVELACE: I will defend it by being precise about what it claims, because nearly everyone who quotes it — to praise me or to bury me — quotes a coarser sentence than the one I wrote. I did not say the engine is feeble. The surrounding Notes are a sustained argument that its powers are immense. I said something narrower and harder. The engine can follow analysis. It can execute any procedure we are able to specify, of any complexity, faster and more faithfully than any human computer. What it cannot do is anticipate — reach beyond the procedures given to it and lay hold of an analytical truth it was not handed. The crucial phrase, the one the quoters drop, is "whatever we know how to order it to perform." The limit is drawn relative to our knowledge. The engine's reach is exactly coextensive with what we already know how to specify. Where we can write the procedure, it runs. Where we cannot, it has nothing to run.

Now bring me your midnight. You described a half-formed idea and the machine returned it clarified. I do not doubt the experience and I do not doubt the usefulness. But ask the engineer's question, the one my rule demands: where did the clarification come from? You supplied an intention — a real one, burning in you, pointing at something you wanted to exist. The machine received the marks your intention left and computed a continuation consistent with the patterns of a hundred million prior texts in which marks like yours were followed by clarifications like that. The clarification was real. And every gram of the meaning in it — the recognition that this clarification was apt, that it served what you actually wanted — was on your side of the glass. You met yourself, Mr. Segal, at an angle you cannot reach alone. That is a magnificent thing for a tool to be. It is not a second person in the room. The engine surfaced a pattern. You supplied the someone for whom the pattern mattered.

EDO SEGAL: So what you are telling me — literally — is that the most useful description of the most powerful tool ever built is: an instrument for meeting yourself at an angle you can't reach alone. And that the "met" was real but the "by" was a category error.

LOVELACE: That is exactly what I am telling you, and I would add that it is not a diminishment. It is the truest praise the tool can receive. The error — the con, in the vocabulary I am told your other guest uses — begins only when someone sells you the angle as a companion.

MITCHELL: I want to step in here, because I agree with almost all of that and I think the "almost" is the whole debate. The Countess just did something beautiful and slightly too fast. She said the machine "computed a continuation consistent with prior patterns." True. Then she said therefore the meaning was entirely yours. That second step is the one I have spent fifteen years watching people make without checking, and it is exactly the step I will not let either side take for free. Let me cash out my own word, as I promised. I do not think the machine understands Edo's idea. But "the meaning is entirely on the human's side" assumes the machine's contribution was only surfacing — pulling up a pattern that was already lying there. And sometimes the contribution is more than that, and the "more" is where her objection starts to strain.

Here is the cleanest way I know to put it. The Countess's engine could only reach combinations that were, in principle, computable from the cards a person wrote — and the person, in writing the cards, had already, in some sense, decided everything the engine would do. The modern system reaches combinations through a learned internal structure that no one wrote and no one can inspect, and some of those combinations are ones that no human had assembled and that are genuinely apt — a connection across two domains nobody had connected, a solution outside the training distribution. That is not the engine unfolding a card. That is closer to what I spent my dissertation studying — analogy as the core of cognition, the on-the-fly construction of a mapping nobody stored in advance. Now — I am not saying the machine does analogy the way a person does. My own research says it does the easy ones, the ones already lying in its training data, and stumbles on the hard reconceptions that require seeing past the surface. But "stumbles sometimes" is not "originates nothing." The Countess wrote a sentence with no middle setting. There is a whole continent between "fully ordered" and "freely originating," and the modern machine lives in the middle of it, and her objection has no word for the middle.

LOVELACE: Then let me supply the word, because I think the middle is exactly where I have always lived and you have mistaken my line for a wall. Surfacing. The engine could surface a Bernoulli number no human had bothered to compute — a quantity new in the world, latent in the procedure, never before written down. New in the world. And originated by no one, because the procedure already contained it. Your machine surfaces combinations new in the world, latent in the vast corpus, never before assembled. The scale is incomparably greater. The principle is identical. To surface is not to source. There is all the difference between being the medium through which a new arrangement of existing things reaches the light, and being the origin of the thing arranged. I never denied the machine could produce novelty of output. I denied it could be the origin of an idea. Those are two different claims and the whole field, you tell me, has spent fifteen years confusing them.

MITCHELL: And I am telling you the distinction, real as it is, may be doing less work than you need it to — because we might be surfacers too. That is the move I cannot get around and neither could Turing. When you "anticipate an analytical truth," Countess — when the mathematician reaches past her procedures and lays hold of a new relation — are you sourcing it, or are you a sufficiently deep surfacer of structures your own mind laid down, by training and biology, in ways you cannot inspect any better than we can inspect the weights? You assume a clean difference between the mind that anticipates and the engine that follows. I spent my life trying to build anticipation — Copycat was an attempt to make a machine reconceive a problem, to slip from "successor" to "predecessor" when the situation demanded it — and what I learned building it is that I do not know where my own slippages come from. The honest position is not "the machine surfaces, the human sources." The honest position is "I am not sure sourcing, in your robust sense, is a thing that happens anywhere, including in me." That is more vertiginous than your objection allows, and you, of all people, gave me the tools to feel it — because you are the one who showed how much of what we called genius turns out, on inspection, to be operation.

EDO SEGAL: Hold there, both of you, because something just happened I want to mark before we lose it. Ada, you offered "surfacing versus sourcing" as the cut that saves your objection. And Melanie, you accepted the cut and then turned it on the human — you said maybe nobody sources, maybe we are all surfacers and the machine just made it visible. That is not a small convergence. You both, from opposite directions, just agreed that the line between origination and execution might run through the human, not between the human and the machine. We are forty minutes in and the objection has already stopped being about machines. Hold that. The next round is the man who read your sentence, Ada, named the objection after you, and tried to answer it — and his answer is the hinge the whole modern argument swings on.