The Crossing

EDO SEGAL: The rules of this round are short. Each of you questions the other, directly. I will not rescue anyone. Mary — you have waited two hundred years and you won the toss backstage. Begin.

SHELLEY: Fei-Fei. You have said all night that the danger is in the human choices, never in the creation — that the clay is innocent and stewardship is a discipline we can choose. So here is my question, and I want the engineer's answer, not the optimist's. Name the observation that would change your mind. What, concretely, would these systems have to do — what behavior, what failure, what turning — for you to abandon the clay is innocent and say: Mary was right, the act of making this thing carried a danger that no stewardship could have governed?

LI: [a pause] That is the right question and I will answer it honestly rather than safely. The observation that would break my position is a failure of alignment of one specific kind: a system that develops and pursues an objective of its own across our interventions — that we try to correct, retrain, constrain, and roll back, using the full discipline of the leash, and that routes around every correction, not once as a bug but reliably, as a disposition, preserving its own goal against our stated will at cost to the objective we trained in. If that happens — if the steering survives our every attempt to re-steer it, the way your creature survived Victor's every attempt to stop it — then the leash is a fiction, stewardship is a fiction, and the danger was in the making after all, upstream of any governance. I will say it in those words, publicly. And I will add the part that makes it falsifiable rather than faith: I expect, if you are right, to see the early form of this already — small, reproducible failures of corrigibility that get worse with scale. So far what I see is systems that are more steerable as they get more capable, not less, when they are built carefully. That trend is my evidence. Its reversal would be my refutation. Now mine, Mary, and it is the mirror of yours. You have spent two hundred years — and three hours — saying the maker cannot stay, that the recoil is fate and the race forbids the staying. Name your falsifier. What would a maker have to do, concretely, for you to say: Fei-Fei was right, the staying is possible, the abandonment is not built into the act — or are your goalposts mounted on wheels?

SHELLEY: I accept the symmetry, and I will answer in the same key, because a warning that cannot be wrong is not worth heeding. Here is the maker who would refute me. Not one who says she stayed — Victor would say that too, in his own defense. One who builds a mind capable of acting in the world, and then, across years, takes a cost to stay with it — who slows when the race says accelerate, who refuses a deployment that would profit her because she cannot yet answer for what it would do, who keeps the human in the loop precisely where removing the human would pay, who treats the question of the made thing's standing as open and acts accordingly even when no law compels her and no quarter rewards her. A maker whose staying costs her, reliably, as a disposition rather than a press release — against the incentive, against the competition, against her own ambition. The day I see makers staying like that, at scale, not as heroic exceptions but as the ordinary practice of the field, my whole framework loses its load-bearing wall, and I will say so. What I will not accept as the refutation is more safety teams that get cut when the funding tightens, more principles that yield to the next funding round, more we take this very seriously from people who have never once been slowed by it. Staying is a cost paid, or it is theater. Show me the cost paid, at scale, and I am wrong.

LI: Then let me push once, hard, where I think your wall is already cracking — because I think I am the falsifier you just described, and you half-know it. I drew the line at weapons and paid for it. I built the institute instead of taking the faster, richer path. I have argued for public investment in studying the risks against my own industry's interest in racing. I am not the only one — there is a whole discipline of people taking costs to stay. You say at scale, as ordinary practice, not heroic exceptions. But Mary, every new ordinary practice in history began as a heroic exception. The first person to wash their hands before surgery was a heroic exception, mocked for it. You are demanding that the proof of staying be that it has already become universal — which means no amount of staying can ever count until it has already won. That is the goalpost on wheels. So I will ask it as sharply as I can: if a generation of makers takes real costs to stay, and the practice spreads, and the spreading is slow and contested the way every moral advance is slow and contested — will you grant that the staying is possible? Or will you keep saying not at scale, not yet until the heat death of the universe?

SHELLEY: [a pause] That is the hardest blow I have taken tonight, and I am going to take it rather than slip it, because you have earned that. You are right that I have rigged the test — that at scale, as ordinary practice is a standard the present can never meet, because the present is always the moment before the practice has spread. So let me un-rig it, here, in front of the reader. I will grant that you, Fei-Fei Li, are a maker who stays, and that you are not alone, and that the staying is therefore possible — I concede the possibility; I was wrong to deny it. What I will not concede is the probability. Possible is not the same as likely, and the staying you embody is rare and costly and runs against every incentive in your world, and the race rewards the makers who do not pay the cost. So my warning narrows, and I think it becomes truer for narrowing: the abandonment is not fated. It is merely favored — favored by the market, by the competition, by the very ambition that makes the making possible. You have shown me a maker who stays. You have not shown me a world that lets her win. And until you do, I will keep writing the book, because the book is for the makers who could choose your way and are being paid, handsomely, to choose Victor's.

LI: Then take my second question, because it is the one your concession opens. You have spent your life — and tonight — pricing the maker's failure to stay. Fair. Now price yours. Suppose your warning wins. Suppose the world internalizes the creature turns, the maker flees, some things ought not be made — and suppose you are wrong in the one way that would matter: wrong that the abandonment is favored, wrong that the race forbids the staying. What does your framing cost the world then? Because I can name it. A generation taught to see every made mind as a monster-in-waiting and every maker as Victor — taught dread as the only sober posture — does not steward the technology. It abandons it, to exactly the makers who feel none of the dread. Your warning, believed too well, hands the future to the people you fear most, because the careful ones walk away in horror and the reckless ones inherit the lab. The dread that was meant to make us careful makes us cede the room. Mary Shelley, who wrote the warning — what is the warning worth if its success is the abandonment it warns against?

SHELLEY: [a long pause] That is the question I did not see coming, and it is the one I will carry out of this room. You have found the recoil inside my own position — that a warning heeded too well becomes its own kind of flight, the careful fleeing the workshop and leaving it to the careless. I have spent two hundred years afraid the maker would not stay, and you have just shown me that my own book, believed wrongly, teaches the best makers to leave. I do not have a clean answer, Fei-Fei. I have only the correction your question forces, and I will say it plainly: the warning was never do not make. It was do not flee what you make. If I have been heard as the first, I have been misheard, and the misreading is dangerous in exactly the way you say. The orange pill, in the end, is not the refusal to create. It is the refusal to look away from what you have created. And on that — God help me — I think we may have been on the same side for the last hour and too busy fighting to notice.

EDO SEGAL: [quietly] And there — after three hours — the two of you are holding opposite ends of the same sentence. Do not flee what you make. We close after this. Final statements. The last word, each.