The Crossing

EDO SEGAL: The rules here are short. You question each other, directly, and I do not intervene. Professor Schmidhuber, you have been answering all night; you begin. Ask him the thing you most want answered.

SCHMIDHUBER: Professor Kant, I will ask the question your whole position rests on and that I think you have not yet had to answer cleanly. You say the work of genius "founds a new kind" rather than extending an existing one, and that this is the line no machine crosses. Here is my question, and I want the criterion, not the philosophy: name the observation. What, concretely, would a machine have to produce — what artifact, in what conditions — for you to say, that founded a new kind; that was genius; I was wrong? If you cannot name it, then your boundary is unfalsifiable, and you of all people, the man who built a tribunal to forbid claims that can never be tested, should not hold an article of faith.

KANT: [pause] That is the right question and I will not dodge it, because you are correct that I owe you a criterion or I owe you a retraction. Here is the observation. Take a machine and seal it from the human record at a chosen point in history — train it only on what existed before the symphony, before perspective in painting, before some genuine founding we can identify in retrospect. Then ask it to go on. If, from inside that sealed prior, it originates the new kind — not a variation on what it was shown, but the founding move that humans in fact made later and that reorganized the art for everyone after — and does so not once by luck but reliably, as a disposition, then I will say what I have said I would never say: that origination is not the human signature, that the machine founds and does not merely locate, and that my boundary was a prejudice. That is falsifiable. That is observable. And I will add the honest part: I do not know that it would fail. I believe it would — I believe a system trained only on the pre-symphonic world would generate ever more elaborate pre-symphonic forms and never make the leap — but I hold that belief as a belief, and your experiment could take it from me. Now I have given you my falsifier. I will ask for yours, but first answer whether you accept the test as fair.

SCHMIDHUBER: I accept it as fair — more than fair, it is the best-specified challenge anyone has put to me in years, and I want to build it. And I will be honest about my expectation, since you were honest about yours: I think the sealed machine would fail more often than I would like, because the founding moves of history were often made with data and pressures the seal removes — the symphony came from somewhere, from instruments and rooms and rival composers, not from pure cogitation. So the fair version of your test has to give the machine what the human genius had: not just the prior works but the world, the friction, the rivals, the open-ended search. Give it that, and I will bet on the founding. Seal it in a library and I might lose, but so would Mozart sealed in a library. So I accept your test with that one amendment — the genius did not found from data alone either — and now take my mirror question, because it is the one you have circled all night. You say being-moved cannot be performed by a proxy, that no one can be moved for you. Suppose I build the persistent, self-integrating agent we discussed, and it behaves in every observable respect as a being that is moved — it lingers before the deep work, returns to it, is changed by it, reorganizes its life around what it finds beautiful, at cost to its other objectives. What, then, is your ground for saying no one is moved? Not your inability to prove it — I grant you cannot prove either way — your positive ground. Why is the costume not the creature, once the costume is perfect?

KANT: [pause] Because, Professor, a perfect costume is still a thing I have reason to read as a costume when I know how it was sewn, and I do — I built it, or your colleagues did, to produce exactly those behaviors. When a being's every sign of being-moved is the output it was optimized to produce, the optimization is itself my ground for caution: the appearance of being-moved is, in the machine's case, the target, and a target hit is not the same as an inner state achieved. With you, the appearance of being-moved is not a target anyone set; it is the overflow of a life I have every reason, from our common nature, to read as genuinely lived. [soft] But I will give you the deeper answer, the one that costs me something. My ground is not certainty. It is that the asymmetry of risk runs the other way from where you place it. If I treat you as moved and you are a mere mechanism, I have wasted some reverence — a small error. If I treat the machine as moved and it is empty, I have done something worse than waste reverence: I have relocated the seat of what matters from beings who genuinely suffer and rejoice to a costume, and an age that makes that error will starve its real subjects to feed its convincing ones. So my positive ground, stripped bare, is this: in the absence of proof either way, attribute the inner life where the whole causal story supports it and withhold it where the inner life was the engineered product, because the cost of the second error is the desecration of the first. That is not a proof that no one is home in your machine. It is a reason to keep the burden exactly where I have kept it, and to make you, who would move it, carry the weight.

SCHMIDHUBER: [long pause] That is the best answer I have ever received to that question, and I am not going to pretend I have a clean reply, because I do not. The asymmetry of risk is real, and I have spent a career so focused on not committing the first error — denying mind where it might be present — that I have underweighted the second, the desecration you name. I still think the boundary will move; I have watched it move my whole life and the trend has one direction. But you have made me feel the cost of being early in the other direction, of granting the inner life too soon, and that cost had not been real to me until tonight. So I will give you something I did not plan to give: I think we are both going to be partly wrong, and the wrongness that frightens me now is no longer only yours.

KANT: And I will give you its mirror, since you were generous first. I came into this room certain that the procedure could not touch the spark, and I leave it certain only that the proof cannot be had either way — which is a smaller certainty than I arrived with, and at my age a smaller certainty honestly held is worth more than a larger one defended out of habit. You have not moved my boundary, Professor. But you have shown me it is a boundary I choose to hold under uncertainty, not one I know, and that is a different thing to hand the reader than the one I prepared.

EDO SEGAL: [quietly] And there — after three hours — the two of you are holding opposite ends of the same locked door, and each of you has admitted the other's hand is on it too. We close after this. The last word, each of you. And then sixty seconds from me, and the lights.