Opening Positions

LOVELACE: Thank you. I want to begin with the thing itself, the mechanism, because every confusion in this subject begins the moment we stop looking at the mechanism and start looking at the impression it makes on us. Mr. Babbage's engine was, in its bones, a perfectly intelligible thing. It had a store, which held quantities — what you would call memory. It had a mill, which operated on them — what you would call a processor. And it was directed by cards, punched cards, borrowed from the Jacquard loom, which specified, operation by operation, exactly what the mill should do and in what order. Feed it one set of cards and it computes the Bernoulli numbers. Feed it another and it does something its designers never imagined. The engine is indifferent. Its behavior is not in its wheels. Its behavior is in the cards.

That was my first insight, and it is the foundation of everything: the separation of the machine from the procedure it carries out. What you call hardware and software, I called the engine and the cards. And once you have made that separation, the objection follows from it as a matter of plain logic. The cards are written by a mind. Every card. There is no operation the engine performs that does not trace back, through the cards, to a human decision. The engine elaborates; it executes; it unfolds the consequences of what it was told with a speed and a reach that humbles its author. But it does not exceed its cards, because there is nothing in it that could. To exceed the cards it would have to anticipate — to reach out beyond the procedure given to it and lay hold of a relation it was not handed. It cannot. It has no pretensions to originate anything. It can do whatever we know how to order it to perform.

Now — I have been told what you have built. I have been briefed on the neural network, on the model trained by gradient descent, on the weights no human wrote. And I want to say, with precision, what I think it changes and what I think it does not. It changes the author of the cards. In my engine, a person wrote each card, knowing what it did and why. In yours, the cards — the weights — were found by a search, an optimization grinding over a corpus until it landed on a configuration that predicts the next word well. No one wrote them. No one can read them. I grant this fully; it is the most interesting thing anyone has told me in a long while. But notice what it does not change. The procedure is still separable from the machine. The behavior is still the unfolding of a configuration. And the configuration was still ordered — not by a hand writing one card at a time, but by a process: the data you chose, the objective you set, the search you ran. You did not write the cards. You wrote the thing that wrote the cards. The chain of ordering is longer, and it is dark in the middle, and you have lost the ability to read it. But it has not been cut. You have built an engine whose author you cannot interview. You have not built an engine that has no author.

So here is my opening, and notice how much of it grants your wonder. The machine you have made is the most general instrument ever conceived. It weaves in any domain you can formalize. It will surprise you constantly, and the surprise is real. But surprise is a fact about the narrowness of your sight, not about the depth of the machine. When you cannot trace the consequence of what you ordered, you call the consequence origination, because you cannot see where it came from. I am telling you where it came from. It came from the cards you did not write but caused to be written. The engine originates nothing. It has only become, at last, opaque enough that you have mistaken your own opacity for its creativity.

EDO SEGAL: Melanie.

MITCHELL: That was the best version of that argument I have ever heard, and I am going to spend three hours disagreeing with the half of it I think is wrong while defending the half I think everyone else gets wrong. Let me start where I stand with her, because it is more ground than my reputation suggests.

I agree the machine follows rules. I agree no amount of fluency is evidence of understanding by itself. I have spent my career saying exactly that, against a field that wanted to call a benchmark score "comprehension" and go home. The Countess and I are on the same side of the war against borrowed words. Where we part is the inference she just drew, and it is a specific, examinable inference, so let me examine it. She says: the configuration was ordered, therefore it originates nothing, therefore the surprise is only our blindness. I want to pull those apart, because the field she does not know yet has spent fifteen years discovering that they do not travel together.

Here is what changed, and it is not a metaphor, it is mechanism. I built my career partly on emergence — on systems made of simple parts following simple rules, with no central control, that produce collective behavior no part contains and no designer specified. An ant colony finds the shortest path to food and no ant knows the map. The behavior is real, it is sophisticated, it is genuinely not located in any component — it lives in the interactions. I studied this in cellular automata that evolved strategies their own programmers could not, at first, even perceive. So when the Countess says "the configuration was ordered," I say: yes, the rule was ordered — reduce prediction error — and from that simple ordered rule, run at a scale that beggars intuition, behaviors emerged that are in no card, that were in no person's mind, that the builders did not put there because they could not have. Capabilities arrive with scale that no one programmed and no one predicted. That is not opacity standing in for creativity. That is emergence, which is a real and studied phenomenon, and it is the thing her objection has no slot for, because her engine — a hand-carded deterministic device — could not do it, and so she built a theory in which nothing can.

But — and this is why I am not on the cheerleaders' side either — emergence is not understanding. This is the whole knife I carry. The ant colony exhibits emergent computation and comprehends nothing. The immune system adapts brilliantly and grasps no immunology. Emergence produces capability without comprehension all over the natural world, and a language model is, in the most precise sense available, another instance of it: enormous network, simple rule, no central controller, astonishing behavior, no one home. So I will not tell you, as Hinton might, that the machine understands. The behavior emerged; the understanding did not follow it into being. The Countess says the surprise is fake because the machine is empty. The hype says the surprise proves the machine is full. I say the surprise is real and the machine is empty, and the fact that those two things can both be true is the single hardest fact of this entire era, and it is the one her 1843 objection, for all its genius, cannot hold.

So my opening is this. She is right that no human wrote the weights and right that the machine does not understand. She is wrong that this makes the surprise an illusion. Something is happening that no one ordered in the sense that matters — not "ordered" the way you order a card, ordered the way you order a rule and then cannot foresee the country it grows. Whether to call that origination is a question about a word, and I want to hold the word open, not slam it in either direction. That is my whole method. Hold the question open, and do the science instead of the slogan.

EDO SEGAL: Before I frame the rounds, one discipline I impose on every long conversation, because it pays off two hours later. Each of you, briefly — what do you envy in the other's position? Not respect. Envy. The thing her side gets to have that yours does not. Ada first.

LOVELACE: I envy her instruments. She has watched these machines. She has run the experiments, seen the failures, held the evidence in her hands. I reasoned my way to the engine from drawings of a machine that never turned a wheel — my entire science of it was conducted in imagination, against a thing that did not exist. She gets to check the calculation, which is the thing I most demanded and least possessed. I had to be right by pure thought, which is a terrifying way to be a scientist, and I was not always right, and I had no way to find out. She can find out. I would have given a great deal for that.

MITCHELL: And I envy her certainty about the boundary. The Countess can stand on a clean line — the machine does only what it was ordered to do, accountability runs back to a mind, the human is the only one home — and push. My whole discipline is the refusal of clean lines. I am the one who has to say "we don't know yet," "the words don't fit," "it's an open question," every single time, while the world wants an answer by Friday. She has the floor under her that I gave up when I decided that intellectual honesty meant living in the uncertainty. There are mornings when her floor looks very restful. I do not get to stand on it. I only get to point at where the others are standing on air.

EDO SEGAL: Two openings and two envies, and the architecture of the evening is already visible. It is not that one of them loves the machine and one of them fears it — they would both tell you to be careful of it. It is that they locate the mystery in different places. Ada says the mystery is in us: the machine is readable in principle, and we have merely lost the thread. Melanie says the mystery is in the system: the behavior genuinely emerged, and is genuinely unread, and may not be reducible to any thread at all. Hold both. We start the rounds at the exact sentence that founded this whole argument — the eleven words from 1843 — and we ask whether they survived the century.