What the Death Cross Measures

EDO SEGAL: Let me bring this down to a kitchen table, because we've been at altitude for an hour and there's a parent out there who doesn't care about microtubules and cares enormously about whether her kid should still become a programmer. In [YOU] on AI I wrote about the software death cross — the moment the cost of building software collapses toward zero because the machine does the implementation, and human value migrates from being able to build a thing to deciding what's worth building. I watched it in Trivandrum: my best engineers became capable of more than all of them together, in a week, because the machine met them in their own language. Marvin, you'd say the machine is climbing toward the deciding too. Roger, you'd say the deciding is exactly the locked room. Marvin, go.

MINSKY: I'd say be careful with "deciding what's worth building," because it's another suitcase. Some of it is taste, some is strategy, some is reading what people want, some is sheer pattern over what's worked before — and most of those, the machine is already learning. The thing people call judgment is not one faculty; it's a stack of them, and the machine is taking the stack one layer at a time, bottom up, the same way it took the implementation. Edo, your engineers didn't keep a magic remainder the machine can't reach. They kept, for now, the top layers — the parts that need a rich world-model the machine hasn't been trained into yet. Watch what happens over a decade. The line between "building" and "deciding" isn't a wall between human and machine. It's a tide line, and it's moving up the beach, and it has not stopped at any natural barrier. I see no reason it stops below the high cliffs, and I've stopped believing in cliffs.

EDO SEGAL: Roger.

PENROSE: Here's where I think my argument earns its keep practically, not just philosophically, and I want the parent to hear it. The amplifier — Edo's word — has a bandwidth. It amplifies whatever it can represent, and it represents the computable. The implementation, the execution, the exploring of a space of options, the generation of variants — all computable, all amplified, spectacularly. But the judgment that sees which option is right — if that involves the non-computable seeing I've argued for, then it's outside the amplifier's bandwidth. The tool makes your engineers faster at execution. It does not make them better at seeing what to build, because it can't process that signal at all. Which is exactly the pattern Edo saw and didn't quite explain: the senior engineers got far more out of the tool than the juniors. On Marvin's "it amplifies everything" theory that's a puzzle. On mine it's obvious — the senior brought the non-computable judgment that directs the amplified execution, and the junior didn't, and the tool can't supply it because it operates in a different domain of physics.

MINSKY: That's a lovely story and the data underdetermines it. The seniors got more out of the tool because they had richer models — more training, more cases in memory, better-organized agencies for evaluating architecture. That's a difference in degree of a computable thing, not evidence of a non-computable thing. Your explanation and mine predict the same observation, Roger — seniors outperform — and you've chosen the one that happens to require your wall. Here's how we'd tell them apart: my theory says a junior, given enough time and the right training, with the tool, builds the senior's judgment, because judgment is learned organization. Your theory says there's a ceiling the junior can approach and never cross, because the senior's seeing is non-computable and unteachable in the relevant sense. That's a real, testable fork, and I'd bet my side of it. We watch juniors become seniors. We watch judgment get built, in humans, all the time. You have to explain that away. I just have to point at it.

PENROSE: I don't have to explain it away — the junior becoming a senior is precisely a human being developing the non-computable capacity, which I've never said is fixed at birth. Humans grow it. My claim is that the machine can't grow it, because the machine is computational and the capacity isn't. You've described humans learning judgment, which I agree happens, and offered it as evidence that machines can learn it, which doesn't follow. The junior and the machine are not in the same situation. One has the non-computable physics in his head and is developing it. The other doesn't have it and never will, however much it's trained.

EDO SEGAL: I want to put one number on the table, because the reader's livelihood is the real subject of this round and we keep floating above it. There's a thing I'd call the Turing trap — the temptation to build machines that imitate humans, displacing them, rather than machines that do what humans can't, augmenting them — and it has an economic shape, not just a philosophical one. If Marvin's right and the tide rises forever, the rational move for every firm is to automate the human out of each task as the line passes it, and the human's wage falls toward the machine's cost. If Roger's right and there's a permanent non-computable layer, the rational move is to pair the human's judgment with the machine's execution, permanently, and the human's wage holds because the thing she supplies can't be supplied any other way. Same factory floor, opposite futures. Which one are we building?

MINSKY: We're building the first one and pretending it's the second, and that's a danger that has nothing to do with whether I'm right about consciousness — which is why I keep saying the metaphysics is the least urgent part of tonight. Even on my view, where judgment is computable in principle, it is nowhere near computed yet, and a firm that fires its juniors today because the machine does the junior work is destroying the only pipeline that produces seniors. That's not a deep error about minds. It's a shallow, catastrophic error about time. The judgment is buildable, eventually, in a machine — and it is built, right now, only in humans, only by the apprenticeship the firm is dismantling for next quarter's margin. So I'd regulate the apprenticeship like a fishery. You don't get to catch all the fish this year.

PENROSE: And on my view the fishery never restocks itself from the machine side, ever, which makes Marvin's "destroying the pipeline" not a transitional risk but the permanent structure of the thing. He says: don't catch all the fish this year. I say: the machine is not a fish and will never spawn one, so the human pipeline isn't a temporary bridge until the machines take over the spawning — it's the only spawning ground there is, full stop, in perpetuity. Which means the case for protecting it is stronger on my account than on his, and again we arrive, from opposite metaphysics, at the same policy. I find that the most reassuring thing I've discovered tonight: the practical answer is robust to which of us is right.

EDO SEGAL: Let me route this through the parent directly, because she's the one I actually answer to. Her daughter is twelve. Marvin, on your view, what should she protect in that child against the rising tide? And Roger, same question — and notice you might give the parent the same advice for opposite reasons.

MINSKY: Protect her ability to learn many ways to do everything. Don't let her become a one-trick mind, because a one-trick mind is exactly what's most replaceable. The child who can only do the thing the machine does best is in trouble; the child who has thirty ways to think, who can switch when one fails, who's built a rich and flexible society in her own head — that child rides the tide instead of drowning in it. And here's the thing — that's good advice whether I'm right or Roger is. Even on Roger's view, you want the kid's whole mind awake.

PENROSE: And I'd give her almost the identical advice and I notice it gladly, because it's a genuine convergence and the parent should take comfort from it: whatever's going on in the metaphysics, the practical instruction is the same. Cultivate the seeing. Protect the friction — the hard, slow, unaided wrestling with a problem that's the only thing that ever built real understanding, in a mathematician or anyone else. My one addition to Marvin's is this: tell her the seeing is not a temporary advantage with an expiry date. Tell her it's hers, permanently, and that the machine's growing power is not a countdown clock against her, because it's running on a different track entirely. Marvin tells the child "stay ahead of the tide." I tell her "you were never in the water." That difference matters at three in the morning, when she's the one who can't sleep.

EDO SEGAL: Mark that — convergence number four, and the warmest one: you give the same homework for the child and disagree only about why. Hold it. Because the next round is the one where the disagreement about why turns out to matter most — education, memory, the apprentice, and the single word "yet" that Roger says is hiding the whole argument.