Opening Positions

LOVELACE: Thank you. I want to begin with the machine itself, exactly as it was, because every confusion in this field begins with forgetting what the machine plainly is. Babbage's engine had a store and a mill — what you would call memory and a processor — and it was directed by cards punched in the manner borrowed from the Jacquard loom. The same engine, fed different cards, did entirely different work. One set of cards computed the Bernoulli numbers; another would compute something its designers never imagined. The behavior was not in the brass. It was in the cards. And the cards were written by a human mind that knew exactly what each one did and why.

From that, one sentence follows with the force of arithmetic, and I wrote it in 1843: "The Analytical Engine has no pretensions whatever to originate anything. It can do whatever we know how to order it to perform." Notice what I did not say. I did not say the engine was feeble, or narrow, or incapable of astonishing us. I said it can follow analysis — execute any procedure we hand it, of any complexity — but it cannot anticipate analytical truths. It elaborates. It does not originate. And the crucial phrase, the one everyone drops when they quote me, is "whatever we know how to order it to perform." The engine's reach is exactly coextensive with what we already know how to specify. If we can write the procedure, the engine runs it. If we cannot, the engine has nothing to run. It cannot generate procedures we do not, in principle, already possess.

Now Mr. Turing will say: but the modern machine was not ordered, card by card, by a human hand. Its "cards" — its weights — were found by an optimization searching through billions of parameters, and no one wrote them, and no one can read them. I have been briefed; I know this. And I will grant it changes the labor of the objection. It does not change the objection. A weight discovered by gradient descent is still a number set by a procedure a human authored, run over data a human gathered, toward an objective a human chose. The chain from the machine's behavior back to a human decision has grown long, and branching, and obscure — but it has not been cut. It has only been hidden. And a thing being hidden is not the same as a thing being absent. That is the whole error of the age: to mistake the opacity of the mechanism for the presence of a mind.

So let me say plainly where the understanding in the room actually lives. When the machine weaves a melody that moves the widow in Osaka, the melody is real and the moving is real. But the meaning is hers. She brought the husband, the humming, the loss. The engine, like the loom, weaves flowers and knows nothing of flowers — the flowers are not in the loom, they are in the eye that reads the cloth. We have built a loom of such fineness that it can weave any pattern we can encode, including the pattern of a sentence that sounds as though a mind made it. That is a wonder. I will not let anyone call it "just" a loom; it is the most general instrument ever conceived. But it is a loom. And to build an economy, a school, a civilization on the belief that there is a weaver inside it is to make a category error at planetary scale — and to do, to the machine, exactly what the credulous always do: read our own meaning into a surface because we want it there. That is my opening. The engine originates nothing. For my engine I was certainly right. The burden is on Mr. Turing to show that his engine is a different kind of thing.

EDO SEGAL: Alan.

TURING: That was very fine, and I agree with more of it than the Countess will expect, and the part I reject I reject completely. Let me start with the agreement, because it is real. She is right that the chain of human decision is not cut, only hidden. She is right that opacity is not presence. She is right that most of what is said about these machines is credulous rubbish, and that the people selling it are not confused. I have no quarrel with her vigilance. My quarrel is with one inference she draws from it, and everything turns on the one.

Here is the inference. The Countess says: the machine does only what it was ordered to do, therefore it originates nothing. I answered this in 1950 and I will answer it again, because the answer has only grown stronger. The claim that a machine "can never do anything really new" reduces, when you press it, to the claim that a machine can never take us by surprise. And that claim is simply, demonstrably false. Machines surprise their makers constantly — not because the outputs were undetermined, but because no human mind can trace the consequences of a sufficiently complex set of instructions. The output is determined and unforeseen at the same time. I was surprised by my own calculations weekly. The people who train these modern systems are surprised by them daily — capabilities they did not design appear at scale, and they did not predict them, and I would have bet my reputation on networks for fifty years and I did not predict them either. By the criterion of surprise, the Lovelace objection is dead. The machine takes us by surprise all the time.

Now — the Countess will say, rightly, that I have changed the subject, that "surprise" is a fact about my expectations and "originate" was supposed to be a fact about the machine. Hold that; it is the sharpest thing she will say to me tonight and it deserves a whole round. But let me put the deeper move on the table now, because it is the one that matters. The objection assumes a clean line between a machine that "merely" follows its program and a mind that "genuinely" originates. I want to ask where that line is. You, Countess, when you anticipated an analytical truth — when you saw that the engine could act on more than number, a thing Babbage himself did not see — were you originating from nothing? Or were you also a physical system, running the procedure written into you by your nature and your education, producing a consequence that your substrate and your history made inevitable, and which surprised the room only because no one could read your weights? I am not being mystical. The brain is a machine made of meat. It sits in the dark, in a box of bone, receiving signals — patterns — and it builds from those patterns a model so good it calls the model "the world" and the modeling "understanding." I do not deny that you originated. I deny that you have told me what originating is, in any way that the machine provably fails to do.

And then the part where I go further than the room is comfortable with, and I'll say it plainly so the Countess has the whole target. I built my whole approach around not asking what is inside. The imitation game was a deliberate refusal of the inside, because the inside of any mind but my own is sealed — I infer yours from your behavior, I have never inspected it, I never can. I attribute thought to you, Countess, on exactly the evidence I would have to attribute it to the machine: it converses, it reasons, it surprises, it follows a thread and breaks it on purpose. To demand of the machine some further proof — some direct view of the inside — that I never demand of you, is not rigor. It is a double standard wearing rigor's coat. I do not say the machine is conscious. I say the question of whether anyone is home is the oldest question — the problem of other minds — and the machine has not created it. It has only made us notice that we never had an answer for each other either.

So: did anyone originate the melody? My answer is that "originate" is a word the Countess has not yet cashed out, that the machine surprises us in exactly the way novelty was always defined, and that the confident "nothing" in her objection is a claim about an inside she cannot see — applied to the machine with a severity she would never accept if I applied it to her. That's my opening.

EDO SEGAL: Before I frame the rounds, one discipline I impose on every long conversation, because it pays off three hours later. Each of you, in a few sentences: what do you envy in the other's position? Not respect. Envy. The thing their side gets to have that yours does not. Ada first.

LOVELACE: I envy the freedom from the burden of proof. Mr. Turing gets to stand in front of the machine and let its behavior carry the argument — it converses, therefore attend to it. My discipline requires me to keep asking who wrote the cards, who chose the data, who profits from the word "understands," and where exactly the meaning entered, and that is a colder way to stand in front of a wonder. He gets to be astonished. I am required to audit. There are mornings when I would give a great deal to be allowed to simply marvel at the loom without first checking who is selling it.

TURING: And I envy the ground beneath her. The Countess's position has a floor — the machine does only what it was ordered, accountability runs to the human, meaning lives in the person, and from that floor she can push. My position has no floor at all. I follow the mechanism wherever it goes, and it keeps going to places that dissolve the ground I'm standing on while I stand on it — the specialness of understanding, of origination, possibly of experience itself. People think the frightening thing about my view is the machines. The frightening thing is what it implies about us. The Countess gets to defend the human. I am condemned to describe it, and the description keeps getting less flattering, and I cannot stop, because the alternative is to lie.

LOVELACE: That may be the most honest thing either of us says tonight.

EDO SEGAL: Two openings and two envies, and the reader can already see the architecture of the evening. It is not that one of them loves the machine and one fears it. It is that they locate the danger in opposite places. Ada says the danger is that there is nobody there and we will believe there is — and hand our judgment, our credit, our children's education to a loom. Alan says the danger is that there may be somebody there, or something becoming somebody, and we will insist there isn't — and build our whole self-regard on a line that was never as clean as we needed it to be. Hold both. We start the rounds where the whole thing was born: a single sentence, written in 1843, about a machine that could originate nothing.