The River, the Death Cross, and the Bank

EDO SEGAL: I'm going to pay the table's toll myself now, with a confession, because we've been on the workbench for an hour and a half and it's time to climb. In my book I described the software death cross — the moment two curves cross, when the capability of the machine rises past the cost of using it, and a single person with the tool can suddenly do what a team did before. I watched it happen to my own profession. And I wrote about the amplifier — the idea that the machine doesn't have a signal of its own, it carries whatever signal you feed it, so the question it forces on every one of us is, are you worth amplifying? Now here's my confession: I cannot tell, when I work with the thing and feel amplified, whether I'm being amplified by a mind that grasped my signal, or echoed by a mirror that has no idea I'm there. And the whole architecture of my book — the climb, the leadership after the inversion — assumes I can tell the difference, or that it matters. Noam, Ilya — for the person on the bank of that river, watching their work get amplified or echoed, does the difference between understanding and mimicry change anything about what they should do Monday morning? Noam first.

CHOMSKY: It changes everything, and I'd say your instinct to confess the uncertainty is the healthiest thing in your book. Here is why the difference matters on Monday morning. If the thing amplifying you is a mirror — and I believe it is — then the judgment, the aboutness, the connection of the words to a world and to purposes, all of that remains yours and only yours, which means it is more precious and more indispensable than ever, not less. The danger of believing the machine understands is that you will delegate the part you can't delegate — you'll outsource the judgment to a system that has the form of judgment and none of the substance, and you'll do it precisely because it's fluent, because fluency is, for us, the reliable sign of a mind. So the person on the bank should use the tool gratefully and ferociously and should hold onto the one thing the tool does not have, which is the understanding of what any of it is for. The mirror can show you your draft. It cannot tell you whether the draft is worth writing. That's the human's, and the temptation of the age is to forget it because the mirror is so smooth.

EDO SEGAL: Ilya, take the same person on the same bank. Does your view tell them to do something different?

SUTSKEVER: Strangely, mostly the same thing, but for the opposite reason, and the opposite reason matters. Professor Chomsky says: hold your judgment, because the machine is a mirror with none. I say: hold your judgment, because the machine is not a mirror — it's a fast, broad, shallow-in-some-ways understanding that is genuinely better than you at many things and genuinely worse at a few crucial ones, and the few crucial ones are exactly where you still live. I've described working with these systems on a coding problem where the model fixes one bug and introduces another, then fixes that and reintroduces the first — it knows enormously more than a junior engineer and yet generalizes worse, fails to converge the way a person would. The machine knows more and understands less, in a specific sense: it lacks the robust, sample-efficient, transferable generalization that a human gets from a handful of examples. So the person on the bank should do what you said, Edo — be amplified, ferociously — but they should plant their flag exactly on the ground where human generalization still beats the machine: taste, judgment, knowing which problem is worth solving, catching the failure the model can't see because it has no stake in being right. That ground is real, and it's where I'd tell my own child to stand. The difference between Professor Chomsky and me is not the Monday-morning advice. It's that he thinks that ground is permanent, because it's the faculty, and I think it's contested — real today, narrowing, and worth defending precisely because it's narrowing.

CHOMSKY: And that's the cleanest statement of our disagreement anyone's made tonight. He thinks the human ground is contested terrain that the machine is advancing on. I think it's bedrock the machine cannot reach because the machine is a different kind of thing. Notice that we give the same advice and tell completely different stories about why it's true. The reader should sit with that, because it means the advice is robust to who's right — hold your judgment either way — but the stakes are not. If I'm right, the judgment is safe and the danger is forgetting you have it. If he's right, the judgment is under siege and the danger is losing it while you're amplified into thinking you still have it.

EDO SEGAL: Then let me put a real wound on the table, because there's a person I worry about more than the master on the bank, and that's the apprentice who never gets to the bank at all. In my book I called it the apprenticeship problem. If the machine does the junior engineer's work, the junior engineer never does the ten thousand hours that turn her into the senior whose judgment we just agreed is irreplaceable. We're amplifying the people who already climbed and removing the staircase behind them. Noam — you taught your whole life. What happens to the formation of a mind when the struggle gets automated away?

CHOMSKY: It's the question that worries me most about all of this, more than consciousness, more than the science, because it's the one that's certainly happening rather than possibly. Understanding is not delivered; it is grown, through exactly the friction the machine is designed to remove. A student who struggles to write the paragraph is building the faculty of judgment in the struggling. Hand her the fluent paragraph and you've given her the product and stolen the process, and the process was the education. I'd put it more strongly than you do, even: the danger isn't only that she won't become the senior. It's that she won't be able to tell that the fluent paragraph she was handed is wrong, because the capacity to tell would have been built by writing it herself. You don't just lose the future expert. You lose the future expert's ability to audit the machine — which is the one thing we've agreed all evening she'd be needed for.

SUTSKEVER: I agree with the worry and I'd add the uncomfortable other half, because I won't only say the reassuring thing. The machine that removes the struggle is also, sometimes, a better teacher than the human had — infinitely patient, available at three in the morning, able to meet the student exactly at her edge. So it's not simply theft of formation; it's a trade, and the terms of the trade depend entirely on whether we design it to make the student do the hard part or to do the hard part for her. The same tool can build the apprentice or hollow her out, and right now the market rewards hollowing her out, because hollowing her out is what feels good and sells. That's not a fact about the technology. It's a fact about the incentives pointing it. And it's fixable, which is why it keeps me up — because fixable things that don't get fixed are the real tragedies.

CHOMSKY: On that, complete agreement, and I'll note it's the same structure as my propaganda point — the technology is a means, and the catastrophe is always in whose interest it's aimed. We keep arriving at incentives. We may arrive there again before the night is out.

EDO SEGAL: That's the most useful thing this debate has produced for an actual human being, and I want to name it as a convergence even though you'll both resist the word: the advice converges, the story diverges. Use the river, hold your judgment — you agree. Why you must hold it — there you split, and the split is the whole tower. Now I want to climb one more floor, into the territory where Noam has spent a second entire career, because the river doesn't only carry our work. It carries our beliefs. And a machine that generates fluent persuasion with no one behind it is a new thing in the history of propaganda. We go there next.