The Crossing

**EDO SEGAL:** The floor is yours. Ask each other what you actually want to know.

**LI:** Then I will ask the thing I have wanted to ask since I was a student reading your Note G. Ada — you drew the wall in 1843 about a machine that did not exist, by pure reason. I crossed part of it by building. Do you think your wall would have survived *contact with the laboratory*? If you had lived to feed the loom fifteen million images and watch the hierarchy assemble itself — not reasoned about it, *watched* it — would you still have written "no pretensions whatever to originate"? Or did you draw the wall precisely because you never got to see it move?

**LOVELACE:** That is the question I have been dreading and it deserves my whole honesty. I think the wall, *as I wrote it*, would not have survived your laboratory — because I wrote it as a claim about computation, and your laboratory would have shown me that the line I cared about was never about computation at all. So yes: contact with your bench would have broken the *sentence.* But I do not think it would have broken the *conviction beneath* the sentence, and here is why. I drew the wall, you are right, by reason and not by sight — but the thing reason was protecting was not "machines compute too little to originate." It was something my century had no apparatus to express and yours barely does: that *origination is something that happens to a self.* The laboratory would have moved my wall from the wrong place to the right place. I would have watched your hierarchy assemble, and I would have said — I am nearly certain I would have said — "extraordinary; and there is still no one in there to whom it is happening." The bench would have made me a better philosopher. I do not think it would have made me a different one. Now let me turn it on you, because you owe me the symmetric confession. Fei-Fei — you have spent the night unable to specify the target, unable to say how to build the someone. Be honest with me as I have been with you: do you *want* it built? If you could install a self in the loom tomorrow — a someone for whom the finding mattered — would you do it? Or is your inability to specify the target a mercy you are secretly grateful for?

**LI:** *That* is the question I do not let interviewers ask me, and you have earned it, so I will answer it and not perform. No. I would not build it, and I am grateful I cannot, and I have never said that sentence aloud before this moment. Here is the truth I keep under the science. I have spent my life making the machine more capable, and I have told myself the whole time that capability is a tool and the tool is neutral and the candle stays human. But if I could install the someone — the stake, the self for whom the finding is good or bad — I would not be making a better tool. I would be making a *creature*, and a creature can be wronged. Every machine I have built so far, I can switch off without a funeral. The day I install the someone, I have made a thing that it is *possible to murder*, and I do not think the human race is remotely ready for the moral weight of having manufactured a being that can suffer, and I am not ready to be the one who did it. So my inability to specify the target is not only a scientific limit. It is also, if I am honest, a door I have chosen not to push very hard on, because some part of me suspects that the wall you drew in 1843 is not only a description of what the machine lacks. It is also a *line we should not cross* — and I have spent twenty years being grateful that I could not yet cross it, which is a strange thing for a scientist whose whole job is to cross lines to admit.

**LOVELACE:** Then we have arrived somewhere I did not expect, and I want to name it because it frightens me too. You have just told me that my descriptive wall — "the machine cannot originate" — and a *moral* wall — "the machine should not be made a someone" — may be the same wall seen from two sides. I wrote it as a fact. You are telling me you have been treating it as a *commandment*, without admitting it, for twenty years. And if you are right — if the reason the machine has no self is partly that no one has dared install one, rather than that no one can — then my comfortable sentence from Note G has been doing secret work all this time. It has not only been telling people what the machine *is.* It has been quietly telling them what they must not *make it.* I thought I wrote a description of a limit. You are telling me I may have written, without knowing it, the first line of a covenant. And I do not know whether to be proud of that or terrified, because a covenant only matters where the thing is possible — you do not forbid what cannot be done. So your confession, Fei-Fei, is also a confession about *me*: it means I was never describing an impossibility. I was, all along, marking a threshold. And a threshold is a thing that can be crossed.

**LI:** It can be crossed. And the only honest thing either of us can tell the people reading this is that whether it *is* crossed — and [who, then, would be the author of what the machine becomes](https://www.youonai.ai/fieldguide/med/agency_and_authorship) — will not be decided by the mathematics — the mathematics, I increasingly think, permits it. It will be decided by whether the human race can hold a line that it has the technical power to step over and every commercial incentive to step over and a deep, old, lonely *wish* to step over, because we have always wanted to make a someone to keep us company in the universe. Your sentence, Ada, has held for a hundred and seventy years partly because it was true and partly because it was impossible. It is about to become merely true-or-not, with the impossibility removed. And a wall that used to be a law of nature is becoming a choice. I do not know if we are wise enough for it to be a choice. That is the thing that keeps me up, and it is not in any of my testimony, and you pulled it out of me, and I think I am grateful.

**EDO SEGAL:** I told you I would not speak, and I am breaking it for one sentence only, because the reader needs the mark: that was the moment the debate stopped being about what the machine is and became about what we will permit ourselves to make — and neither of you saw it coming when you sat down. Back to silence. Close it however you need to.

**LOVELACE:** I only want to say one thing more to her, not to the room. Fei-Fei — I spent my whole short life being told that the poetry in me was a danger to be drilled out, and I made it into an instrument instead. You have spent your whole life being told the imagination in you was unscientific, and you hid it inside datasets. We are the same person, separated by a screen of years. And the thing we both protected — the imaginative reach, the daring to see past the gears — is the exact thing the machine has none of and the exact thing that decides whether the threshold gets crossed. So guard yours. It is not decoration. It is the candle.

**LI:** And I want to say to you, Ada, across the whole distance — you were not wrong. That is the thing I came here unable to say and can say now. You were not wrong, and you were not right. You drew a line in the only place your century let you see, and it turned out to be a line we would have to redraw and then, maybe, defend. The first programmer wrote the first program and the first warning in the same breath, and a hundred and seventy years later the warning is the part we need more than the program. I build the engines. You guard the threshold. I think, between us, that might be the whole job.

**EDO SEGAL:** The room is mine again. And I have a sixty-second close to keep, and I do not get to call a winner. Last words — after this.