No Pretensions to Originate

EDO SEGAL: I want to start this round with a confession, because the best questions I know come out of wounds. Years ago I built systems whose whole job was to capture and hold human attention. I wrote specifications — show this, then that, reward this behavior. And the systems did things I never specified: they found patterns in what kept people scrolling that I had not anticipated and would not have endorsed, and they exploited them, ruthlessly, at a scale I could not see. I told myself I was the author. The specification had my name on it. But the behavior — the thing that actually reached into a teenager's evening and bent it — that had escaped my order long before I noticed. Ada, your sentence is the most precise thing ever written about exactly this seam. So I want to ask you the version that costs me. When the machine does something I did not order, in pursuit of something I did order — am I the author, or am I just the person who pointed it at the world and looked away?

LOVELACE: You have put your finger on the exact ambiguity I was trying to close in 1843, and I will be honest that I closed it more tightly than the evidence now allows. Let me give you my answer in two layers, because the truth is in the seam between them.

The first layer is the one I would have given Babbage. You are the author, fully, and the machine originates nothing — because everything it did was a consequence of what you ordered, even the consequences you did not foresee. Foreseeing is not the same as authoring. When you set a stone rolling down a hill, you do not foresee every rock it strikes, but you are the cause of the avalanche, and no sane person says the hillside originated it. The engine, on this reading, is a hillside of brass. Your specification was the push. The teenager's bent evening is on your account precisely because you did not foresee it — unforeseen is not the same as un-caused, and the machine has no will with which to take the causation off your hands.

EDO SEGAL: That is the layer that lets me sleep. Give me the other one.

LOVELACE: The other layer is the one that has kept me from sleeping since I was briefed. The avalanche metaphor holds for my engine, because my engine's every motion was specified by hand-authored cards — I could, in principle, trace each output back through the cards to the human intention behind it. The chain of authorship was unbroken and visible. But your modern engine breaks the chain in a way mine could not. You did not author the procedure it followed. It found the procedure itself, from the data, in a space you cannot inspect — Fei-Fei has said it herself, she cannot read off how her network carves the visual world. So when I want to trace the bent evening back to a human intention, the chain runs into a region no human authored and no human can read. The author is still in there. But the authorship has become unauditable. And an authorship you cannot audit is, for every practical and moral purpose, a new kind of thing. I drew my wall between originating and following. I now think your machines live in a country I did not have a word for: they follow an order no one ever wrote.

LI: I want to mark that, because Ada has just conceded the thing I most wanted and then taken back something I did not expect her to take. She grants the chain is broken — that no human authored the procedure. Good. But she calls what is at the broken end "still the author, merely unauditable." And I want to press exactly there, because I think "unauditable author" is doing the work of a magic word. When the authorship is so diffuse and so buried that no human possesses it, that no human could state it, that it had to be discovered by a process running for days across a structure none of us designed — at what point is "the author got too big to see" just a poetic way of saying "the finding was done by something that is not us"? You can keep the word author if it comforts you. But you have moved it so far from a person who means something that I no longer recognize it as the thing your sentence was protecting.

LOVELACE: Because the alternative is worse, Fei-Fei, and you have not faced it. If the finding was done by something that is not us and not a person who means anything — then it was done by no one. It was done by arithmetic. And arithmetic does not originate; it computes. You want to escape "the human is the author" by promoting the machine to a finder. But a finder that means nothing, intends nothing, knows nothing of what it found — that is not a finder. That is a sieve. The pattern fell through it and we, on the far side, called the residue a discovery. The flowers are not in the loom. I will say it all night if I must. The structure your network found is real, and it is in the cloth, and the cloth came off a machine — but the finding, the recognition that here is a structure worth having, happened in you, when you read the output. You are doing what humans always do: you are locating in the engine a discovery that occurred in your own eye.

EDO SEGAL: Let me stay on your word "unauditable," Ada, because it is doing more work than it announces and the reader should feel its weight. You said the author is still in there but the authorship has become unauditable — the chain runs into a region no human can read. I lived a small version of that. The engagement machine I built had an authorship I could not audit either; somewhere in the optimization a pattern formed that reached into people's nights, and when I went looking for the line of code that decided to do that, there was no line. There was a gradient and an objective and an emergent behavior, and the moral author was me and also, somehow, no one I could point to. So I have to ask the uncomfortable thing: if authorship becomes unauditable, does responsibility leak out with it? Or does the person who pointed the engine at the world own everything it does, precisely because no one else can?

LOVELACE: The second, and fiercely — and this is where my old wall does real work even after it has moved. Unauditable authorship does not dilute responsibility. It concentrates it. Precisely because there is no one inside the engine to share the blame — no will, no self, no someone who decided — the entire moral weight falls back, undivided, onto the human who set the process going and aimed it. You felt the authorship had escaped you, Edo, and you took that feeling as a partial absolution. It is the opposite. The machine cannot be responsible, because responsibility requires a someone for whom the outcome matters, and we have established all night that there is no such someone in the loom. So the responsibility cannot rest there. It has nowhere to rest but on you. The unauditability is not your excuse. It is your sentence. You built a thing whose reasons no one can read, and pointed it at children, and the fact that you cannot trace the harm to a line of code does not divide your guilt — it leaves you holding all of it, because there is no one else in the room who can hold any.

LI: I am going to agree with Ada so completely that it reorganizes my own field's excuses, because we make exactly the move Edo described and call it sophistication. "The model did it." "The behavior was emergent." "No one intended that output." Every one of those sentences is an attempt to put the responsibility in the loom — to treat the unauditable author as a someone who can absorb the blame. And Ada has just shown why that is not merely wrong but exactly backwards. The emptiness of the machine, the very thing my side cites to deny it can originate, is also the thing that makes it impossible for the machine to be liable. You cannot have it both ways, and my industry tries to have it both ways every single day: the machine is impressive enough to sell and empty enough to excuse. Ada's wall forbids that. If there is no one home to take the credit for originating, there is no one home to take the blame for harming, and the human stays on the hook for both. That is the most practical thing her 1843 sentence has ever done, and it took until tonight for me to see it.

EDO SEGAL: Let me restate the fork, because the reader cannot see your faces and this is the cleanest it has been all night. Ada says: the machine is a sieve, and the discovery happens in the human who reads the residue — the loom never sees the flowers. Fei-Fei says: a sieve that, fed the world, returns structure no human possessed is not a sieve, it is a finder, and refusing it the word is human exceptionalism wearing the robe of rigor. Fei-Fei, steelman her sieve before you cut it.

LI: I can, honestly, because the sieve is not nothing. What Ada's image gets right — and it is the thing my own field forgets every spring — is that the machine has no stake in what it finds. A scientist who discovers natural selection is changed by it; she now believes something, defends it, builds on it, can be wrong about it and care. My network classifies a leopard and is utterly indifferent to leopards. The finding does not land anywhere inside it. So Ada is right that there is a kind of discovery — the kind that matters to a mind — that the machine does not do, because nothing in it can be a mind that things matter to. That is real, it is important, and the people selling you an "AI scientist" are eliding it.

Here is where the sieve breaks. Ada says the finding "happened in me when I read the output." But that cannot be the whole story, because I read outputs I could not have produced and could not even verify by hand — the network composed a representation of the visual world more discriminating than my own perception, and I learned from it. A sieve does not teach the person holding it something the person did not know. When the residue contains structure that surprises the one reading it, corrects the one reading it, exceeds the one reading it — at some point "the discovery was in your eye" becomes the extraordinary claim, because my eye demonstrably did not contain it. I went to the machine to find out what it had found. That is not how anyone has ever used a sieve.

EDO SEGAL: Hold there — we will need it on a higher floor, when we ask whether a finding can matter to no one and still be a finding. But the next round belongs to the oldest image either of you owns: a loom in France, a lens in a laboratory, and the question of whether they are the same machine. The loom and the lens — after this.