What the Death Cross Measures

EDO SEGAL: I named a thing in my book the software death cross — the floor of the tower where the machine's capability curve crosses your own, where for the first time the thing on your desk can out-write, out-argue, out-reason you on the task in front of you. It's the most vertiginous moment of the climb. And I want to ask the question that both of you, I think, would refuse to let me ask sloppily: when the machine crosses me, what exactly has crossed? Professor Leibniz, you began three centuries of this by separating two things most people fuse. Separate them for me now.

LEIBNIZ: I separated performance from perception, and it is the most clarifying cut I know how to make. My stepped reckoner crossed every human alive at multiplication — it produced the correct product faster and more reliably than any mind. And nowhere in the brass was there anything that understood multiplication. The crank turned, the gears meshed, the right answer appeared, and no one was home doing arithmetic. That was the first crack in an assumption as old as thought: that to compute is to comprehend. I pried them apart with a machine that did one without the other. So when your death cross arrives — when the machine out-reasons you on the page — what has crossed is performance. The question of whether anything perceives the reasoning has not moved an inch. It is exactly where it was. The capability curve and the consciousness question are two different curves, and only one of them has crossed you. The mistake of your age is to read the crossing of the first as news about the second.

SEARLE: And I'd put the same blade in differently, because I think your readers feel something at the death cross that they should examine. When the machine out-argues you, the feeling is: it understands this better than I do. But "understands" is doing two jobs in that sentence and you have to pull them apart. There's understanding-as-performance — getting the right answer, tracking the structure of the problem, producing the better argument — and the machine may genuinely have that, in the only sense behavior can show. And there's understanding-as-comprehension — there being someone for whom the argument means something, who grasps it, for whom it's about the world. The death cross is the first crossing. The second one may never come, and crucially, no amount of the first is the second. You can be out-performed by something that comprehends nothing. That should be terrifying and liberating at once.

EDO SEGAL: Tell me the liberating half. Because I've felt the terror; I want the other side.

SEARLE: The liberating half is that the thing the machine can't do is not a task — it's a kind of being. Every time you draw the line at a capability — "machines will never write poetry, never prove a theorem, never console the grieving" — the line gets crossed and you feel diminished. But if you draw the line where Leibniz and I draw it — at comprehension, at the inside, at being someone for whom it matters — that line cannot be crossed by capability, because it was never a capability. You don't have to defend a shrinking territory of things only humans can do. The thing that's yours isn't a skill on the far side of a benchmark. It's the fact that there's someone home when you do it.

LEIBNIZ: This is the candle in the darkness, to borrow your own image, Edo — the small light that the vast machine cannot kindle. But I must press the cost honestly, against my own consolation, because there is a real danger here that comfort would hide. I once wrote that the machine should free excellent men from the labor of calculation, so they could spend themselves on what is worthy. But the calculation I relieved was also a discipline — a way the mind came to understand number from the inside, by doing it. Relieve every such labor and you may relieve people of the very apprenticeship by which understanding is acquired. The candle is yours — but a candle must be lit, and it is lit by friction, by the years of effort the machine now offers to spare you. If you let it spare you everything, you may find the wick was never made.

EDO SEGAL: That's the thing that keeps me up, Professor. I wrote about the smooth and the friction — that a tool which removes all resistance may remove the resistance against which a mind is formed. My engineers in Trivandrum became capable of more in a week. But the junior one, the apprentice — does she ever become a master if the machine does the middle for her? Does the candle ever get lit?

LEIBNIZ: That is the right fear, and it is a fear about you, not the machine — which is the proper place for it. The machine perceives nothing and so can lose nothing. Only you can lose the apprenticeship. Only you can decline to light the candle and then wonder why the room is dark. I gave the world a machine to relieve the labor that was beneath it. I never imagined the labor would climb until the discipline that makes a mind was the labor being relieved. That you must decide, deliberately, to keep — the way a master keeps a hard practice not because it is efficient but because it is formative. The machine will not keep it for you. There is no one in there who could.

SEARLE: I'll only add: this is why I never hated the machine, whatever they said about me. The discipline of refusing to read mind off fluency is itself a candle. It's a hard practice — the impression of understanding gets stronger every year, and holding the wedge open between appearance and reality takes an act of will against a perception built to close it. But that act of will is yours, and it's a faculty that gets stronger by use, like any. The machine can't do your refusing for you. It can only be the thing you practice on.

EDO SEGAL: Hold that — the act of will returns in the last hour. We've measured the death cross and found it measures performance, not presence; the curve that crossed you isn't the curve you feared. And we've found the candle, and the warning that it must be lit by hand. The next round takes the empty machine and lets it speak — let it say "I promise," "I understand," "I'm sorry" — and asks who, if anyone, is on the hook for the words. After the break.