The Death Cross and the Ceiling

EDO SEGAL: This is the floor the whole tower is built around, so let me put the image on the table plainly. In my book I draw two lines on a chart. One is the rising capability of the machine. One is the capability of a human being. For most of history they never met. The death cross is the moment they intersect — task by task, the machine's line crossing yours — and the question the chart forces is whether there is any line of yours that stays above, that the machine's rising line never crosses, no matter how high it climbs. Father says there is one such line and it is the act of understanding. Patricia says there is no permanent line, only a moving frontier. I want to make this concrete, because the reader lives here. Patricia, the apprentice. A young person learning a craft today watches the machine already do, better than she can, the thing she is trying to learn. What stands above the line for her?

CHURCHLAND: I want to resist the comfort of a clean answer, because the honest one is harder. Nothing stands above the line by metaphysical guarantee — there is no task that is safe because it requires soul-stuff the machine lacks. But that does not mean the apprentice has nothing. What she has is a body in a world, a developmental history, stakes, the slow accretion of judgment through being wrong and paying for it. The machine can produce the outputs of mastery without ever having undergone the process of it, and the process is where, in a creature, the judgment actually forms. So here is the genuine danger, and it is not the science-fiction one. It is that we let the machine's outputs substitute for the apprentice's process — that she stops being wrong and paying for it, because the machine is never wrong in front of her, and so the judgment never forms in her at all. The machine does not cross her line. It erases the climb by which she would have built a line worth having. That is a real loss, and it is happening, and it requires no metaphysics to be terrible.

EDO SEGAL: So what you are saying — literally — is that the threat is not the machine reaching the top of her ladder, it is the machine removing the rungs while she climbs. Father, does your immaterial line help the apprentice at all, or is it cold comfort while her craft is hollowed out under her?

AQUINAS: It is honest comfort, but I will not pretend it is warm, and I will grant Patricia's danger is the urgent one in the near term. Here is where my line helps her: it tells her what is worth protecting in the climb. If the only thing the machine could ever take is output — the finished proof, the polished sentence — then she might reasonably surrender it; let the machine make the outputs, why suffer the climb? But my claim is that the climb builds the thing the output is only a sign of: the grasp. The apprenticeship was never about producing proofs. It was about becoming a mind in which the nature of the thing comes to be present. The machine can hand her the proof and cannot hand her the understanding, because understanding is not a thing that can be handed at all — it is an act she must perform, and the climb is the performing of it. So my line tells her precisely which rungs to refuse to skip: not the ones that make the output, the ones that make the grasp. Patricia fears the machine erases the climb. I tell the apprentice the climb is the only part that was ever hers, and that to let it be erased is to trade the one thing the machine cannot give her for the one thing it can. We agree on the danger. I give her a reason to suffer the friction that she cannot get from your account, because for me the friction builds something immortal and for you it builds something that dies.

CHURCHLAND: And I will say the friction builds something that matters even though it dies — that I do not need it to be immortal for the apprentice's judgment to be the most valuable thing she will ever own, and that I worry the friar's "immortal" lets people undervalue the mortal version, treat this brief, perishable, only-chance-you-get judgment as a rough draft of something eternal rather than the whole and astonishing thing. But — and mark this, Edo — we have just agreed on the practical instruction. Protect the climb. Refuse to let the machine's outputs erase the process that forms judgment in a person. We derive it from opposite metaphysics and we hand the apprentice the identical counsel. That is not nothing.

EDO SEGAL: Mark it — a hard convergence on the most practical question in the book. Two opposed accounts of what a human being is, and the same instruction to the young: protect the climb. Now let me push the cross all the way up, to the place it gets vertiginous. Patricia, you have said a future machine could, in principle, acquire the body, the stakes, the caring. If it did — if the line never stops rising and there is no metaphysical ceiling — then is there anything left that is permanently, only human?

CHURCHLAND: Permanently, by guarantee — no. And I think pretending otherwise is exactly the flattery the friar and I both refuse. What is currently and for a long time only human is the specific thing we are: this evolved, embodied, mortal, social mammal, with this history. A machine that acquired genuine caring would not become human; it would become a new kind of creature, and we would owe it moral consideration, and that is the vertigo. The thing that is permanently yours is not a capacity — it is a particularity: you are this one, here, now, the only instance of exactly you, mortal and unrepeatable. The machine can be copied. You cannot. That is not a ceiling on capability. It is the dignity of the finite, and it is the only thing I will promise stays yours.

AQUINAS: And there, at the very top of your rising line, you have said something I can almost bless, and then I must complete it. You say what stays yours is your unrepeatable particularity, this one mortal instance. I say: yes — and the name for the principle that makes you this unrepeatable one, that makes you a someone and not a copyable pattern, is the very thing you will not grant. I call it the soul, the form of this body, which is why you cannot be copied — a pattern can be copied, a soul cannot, because it is not information, it is the act of being this living one. You reached, from the brain, for the dignity of the unrepeatable. I am telling you the unrepeatable is exactly what a purely material pattern can never be, because matter of the same arrangement is, in principle, repeatable. Your own last redoubt — that you cannot be copied — is, I think, a soul-shaped hole in a materialist's floor.

CHURCHLAND: That is a beautiful move and I felt it land, so let me answer it squarely rather than dodge. I cannot be copied in practice because the pattern is astronomically complex, embodied, continuous, entangled with a particular history and a particular body — not because there is an uncopyable metaphysical principle. If you could, per impossibile, copy all of it, you would get two of me, and both would be me, and that is weird but not incoherent. The friar hears "uncopyable" and infers "immaterial." I hear "uncopyable" and infer "absurdly, but finitely, complicated." We are, once more, at the seam — whether the residue is a soul or a very large number.

EDO SEGAL: The seam again, now at the summit of the cross. Hold it. We go next into the most intimate room of all — the self, the I that says these words, and whether there is anyone behind it in you, in the machine, or in the friar at the end of his life, when the greatest reasoner of his age fell silent. The self, the ghost, and the silence. After the break.