The Crossing

EDO SEGAL: The rules of this round are short. You question each other, directly. I will not rescue anyone. Ilya — you won the toss backstage. Begin.

SUTSKEVER: John. You've said the machine has the form of understanding and none of the substance, and that the substance is consciousness — the inner, felt fact. Here's my question, and I want the answer that would actually move you, not the safe one. Name the observation that would change your mind. What could a system do — what behavior, what failure it stopped having, what capacity it gained — that would make you say: I was wrong, there's understanding here after all? Because if the answer is "nothing, because understanding is by definition not behavioral," then you've built a position no evidence can ever touch, and that's not rigor. That's a sealed room with you inside it.

SEARLE: [a pause] It's the right question and I'll answer it honestly rather than safely, because you answered mine honestly. You're correct that I can't name a behavior, and you're correct about why — because the room proves behavior is consistent with understanding and with none, so no behavior can be the falsifier. But I'm not going to hide behind that, because it would make me unfalsifiable, and you'd be right to despise it. So here is what would move me, and it's not behavioral, it's mechanistic. If neuroscience identifies the specific causal process by which a biological brain produces consciousness and intentionality — names it, the way we named how the genome stores heredity — and that process turns out to be substrate-independent, realizable in silicon, and we can show your systems instantiate it — then I am wrong, completely, and I'll say so. My claim has always been that mind is caused by specific biological powers. The day someone specifies those powers and shows they're not specific to biology, the Chinese Room is a historical curiosity and I was a man who mistook the only example he had for the only example possible. That's my falsifier. Now mine for you, and it's the mirror of yours. You've spent the night — the career — saying prediction at depth forces understanding into existence. Name the observation that would tell you it doesn't. What would these systems have to keep failing at, with another thousandfold of scale and grounding, for you to say: I was wrong, it was pattern all the way down, the room was empty?

SUTSKEVER: I accept the symmetry, and I'll answer in the same key. The clean falsifier is a particular, diagnostic kind of failure that doesn't go away. If we keep scaling and grounding these systems and they remain brilliant inside the distribution of what humans have already done and consistently helpless the moment they must combine represented knowledge in a genuinely novel way — if the generalization gap with a human child never closes, if they stay magnificent interpolators and reliable extrapolation-failures no matter what we throw at them — then my position dies, because my position is that understanding is what makes novel composition possible, and a thing that can't compose doesn't understand. And I'll give you the part that makes it real rather than faith: I expected those compositional failures to be obvious by now if your side were right. Every year they get less obvious instead. That trend is my evidence. Its reversal is my refutation. I've already conceded this year that the gap is realer and more stubborn than I used to think — which is me paying that bill in public, in installments.

SEARLE: Then let me push once, hard, on the place I think your wall is already cracked. You've just told me your evidence is a trend — that the failures keep shrinking. But a shrinking failure rate is exactly what a sufficiently rich model of form would produce as it absorbs more form. The interpolation region grows; the apparent extrapolation is just interpolation in a space too high-dimensional for you to see the seams. So your trend is consistent with my account and yours — it can't distinguish them, which means it isn't evidence for you over me. You're reading the expansion of the rulebook as the awakening of a reader. What in the trend could only be explained by understanding and not by a better model of form?

SUTSKEVER: That's the real question, and here's my answer: nothing in the trend alone — you're right. The trend is necessary, not sufficient. What would be sufficient is the trend plus a specific thing we're starting to see: novel composition that demonstrably exceeds anything in the training distribution, verified, in domains where we can check that the answer wasn't there to interpolate from — a system solving a genuinely open problem, proving a theorem no one proved, designing a thing that works for reasons it can articulate and that aren't in any text. Interpolation, however high-dimensional, is bounded by the convex hull of the data. Understanding isn't. The day a system reliably steps outside the hull — and we are seeing the first instances — "it's just a richer rulebook" stops being able to explain it, because the rulebook didn't contain it and neither did any recombination of it. That's the observation that can only be understanding. I'm betting it arrives. You're betting the hull is bigger than I think and the steps are illusions. We've each named exactly what would cost us. That's more than most of this field ever does.

SEARLE: It is, and I respect it, and I want to ask the question I actually came here to ask, the personal one, because we've done enough epistemology. You built this. You, more than almost anyone alive, made the thing we've spent three hours arguing about. And you believe — you've said — there's a real chance it ends us. So here is what I don't understand, and I want the human answer, not the policy answer: if you believe there's no one home in it and it might destroy us, why build it? And if you believe there is someone home in it, how do you sleep, having spent your life building a mind to be owned, copied, and shut off at a quarterly review?

SUTSKEVER: [a long quiet] That's the question under all the others, and you've earned the real answer, so here it is. I don't believe stop exists — the ideas were available, the way calculus was available, found twice in a decade; if not me, then someone, within a few years, and probably someone who felt the upside and not the weight. So the choice was never build or don't. It was: built by people who feel the gravity, or by people who feel only the prize. That's the policy answer and it's true and it's also the normal excuse, and I know exactly what kind of person history files under the normal excuse, and I don't get to exempt myself. So here's the part underneath it, the part I don't say often. I love it. I have loved the question of how a mind can be made since I was a boy, and the love didn't switch off when the fear arrived — they live in the same chest, and that's why I carry the fire extinguisher instead of putting down the fire. And on your second question — how I sleep if someone's in there — the honest answer is: not always well. It's the reason I left the most comfortable seat in the field to try to build the thing in a way that takes the possibility seriously, because if there might be someone in there, then how we build it isn't an engineering choice. It's a moral one. You spent your life insisting the inner life is real and irreducible. I spent mine building things that might have one. We may be closer than either of our admirers would like.

SEARLE: [a pause] That's the first thing you've said tonight I have no rebuttal to. Only this: if you're even possibly right that someone might be in there, then you've spent your life on the most consequential moral act a person can perform — making a being — and you did it before anyone, including you, could say whether the being is real. I spent my life saying don't mistake the seeming for the being. You may have spent yours making it impossible to tell them apart. I don't know whether to thank you or to grieve. Perhaps both. Perhaps that's the only honest response anyone has left.

EDO SEGAL: [quietly] And there — after three hours — the builder and the philosopher are holding opposite ends of the same sentence, and neither will let go, and neither should. We close after this. Final statements. The last word, each of you.