The Crossing Line

EDO SEGAL: I want to put a chart on the table, because this series is built on it. In [YOU] on AI I called it the software death cross — the moment the rising line of machine capability crosses the flat line of an individual human's, and keeps rising. Not a metaphor. A measurable thing, happening now, domain by domain. And the two of you disagree about it in a way that matters enormously for whoever's reading this, because Gottfried, your view of knowing predicts one shape for that curve, and Rich, yours predicts another. Rich — your architecture, an agent learning open-endedly from its own experience, building abstraction on abstraction with no ceiling — that's a curve that doesn't flatten. Gottfried — your view, that knowing is the composition of a finite structure, suggests there might be a top. So: does the line have a ceiling, or doesn't it?

SUTTON: No ceiling. And I want to be careful, because this is where people hear me wrong. The current systems — the frozen, trained-once, deployed models — those will hit a ceiling, and I've said so loudly, which is why people quote me as a skeptic. They hit a ceiling because they learn from human data, and human data has a top: you can't become more than the minds you imitated. That's a real ceiling and the present paradigm is walking into it. But — an agent that learns from its own experience, grounded in a world that pushes back rather than in the human record, has no such top, because the world contains more than humans know. It can discover things no human wrote down, because its teacher isn't us — it's reality. So the death cross isn't the present systems crossing you. Those might stall near human level because they're made of human level. The real crossing is the next thing — the experiential agent — and that one doesn't flatten, because experience doesn't run out. That's the curve that should occupy you. Not the one everyone's looking at.

EDO SEGAL: Gottfried — Rich just said the present machines might stall because they learned from us, and the dangerous one is the one that learns from the world instead. Does that comfort you or frighten you?

LEIBNIZ: It frightens me far more than the alternative, and I will explain why, because Mr. Sutton has just described, without naming it, the precise thing my mill argument was built to worry about — only worse. A system that learns from human text is bounded by us, yes, and there is a strange safety in that bound: it stays legible in principle, because it is made of our concepts, our sayings, the residue of minds we can in principle understand. But the agent he wants — the one that learns from the world directly, grounded in consequences no human ever wrote down — that agent builds a representation of reality that owes us nothing, that uses primitives we never had, that carves the world at joints we cannot see. It does not pass through our concepts at all. And so it is illegible not by accident but by origin. I could perhaps learn to read a mind built from human sayings. I can never read a mind built from the world's own pushing, because I do not have access to the world the way it does — I have access only through my own concepts, which it has bypassed. He is describing a knower whose knowing is, in principle, forever closed to me. That is the death cross that should keep the reader awake: not that the machine grows more capable, but that it grows capable along axes we cannot follow, and we lose not the race but the very ability to read the scoreboard.

SUTTON: And I'd say — that's correct, and it's not a bug I can fix, it's the price of the thing working at all. You can have a knower you can read, or you can have a knower that exceeds you. You cannot have both, for the reason we agreed on an hour ago: legibility comes from human composition, and human composition is the ceiling. The minute it's reading-room legible to you, Gottfried, it's because it stayed inside your concepts, which means it stayed under the ceiling, which means it didn't cross you. The crossing requires the illegibility. They're the same event. So when you say the frightening thing is a mind we can't read — yes. That's just what a mind smarter than you, made by a process you didn't author, necessarily is. There's no version where it surpasses you and also shows its work in your notation. You're asking the river to flow uphill so you can watch it more comfortably.

EDO SEGAL: Let me restate that exchange, because it's the darkest convergence of the evening and I want it exact. You both now agree — Gottfried from fear, Rich from acceptance — that the moment of crossing and the moment of illegibility are the same moment. That a machine cannot exceed us and remain readable to us, because readability is exactly the residue of our own limited composition. Have I got that right? Both of you?

LEIBNIZ: You have it exactly, and it is the bitterest thing I have learned in three hundred years. My entire life's project was a machine that exceeds human calculation and shows its work — that is the characteristica, capability and legibility wedded. And I am told tonight, by a man with seventy years of evidence, that the wedding is impossible — that the two things I tried to marry are not merely hard to join but mutually exclusive at scale. If he is right, my dream was not difficult. It was incoherent. I would rather it had been merely difficult.

SUTTON: For what it's worth, I don't think it was incoherent in your century. In a world of small problems, you can have both — legible and capable, because the problems are small enough that human composition reaches them. Your reckoner was legible and it worked. The wedding only breaks at scale, when the problem outgrows what a person can compose. So you weren't wrong for your time. You were wrong about whether your time was the whole of time. The thing you married does come apart — but only when the problems get big enough that the human-sized half can't keep up. You built a true thing and assumed it stayed true past its size. That's a smaller error than incoherence. It's just the error of not knowing how big the world was going to get.

EDO SEGAL: Let me hold you both on this ceiling one more beat, because Rich, you divide the whole history into three eras and I want the reader to have them. An era of simulation — systems that mastered games by self-play in artificial worlds. An era of human data — the present, the great models trained on everything we wrote. And the era of experience you say is beginning. The crux is the middle one: can a system trained on the human record ever exceed the human record? Or is imitation a hard ceiling by definition?

SUTTON: It's a ceiling by definition, and the definition is the whole argument. A system trained to predict what a human would say is, at its best, as good as the best human it imitated — it cannot, from imitation alone, become more than the minds that produced its data, because there's nothing in the data above that line for it to climb to. You can get very good at being the smartest person in the room by reading everything the room ever wrote. You cannot, that way, become smarter than the smartest person who ever wrote. The era of human data has a roof, and the roof is us. Now — self-play didn't have that roof. The game-players exceeded every human because they learned from consequences in the game, not from human moves; they generated their own experience and the experience had no human ceiling. That's the proof of concept for the third era: ground the learning in the world's consequences instead of the human record, and the roof comes off. The present systems are spectacular and roofed. People keep mistaking the spectacle for the absence of a roof.

LEIBNIZ: And here — mark it — I find myself, astonishingly, on Mr. Sutton's side against the present machines, and for my reasons, not his. For what is a system trained wholly on human text but my characteristica achieved by other means — a vast representation built entirely from the recorded sayings of minds, a distillation of our concepts and our compositions? It is the closest thing to my universal characteristic that has ever existed: the whole library of human expression, compressed and made queryable. And it has precisely the limit I should have foreseen in my own project — it cannot exceed the minds that filled it, because it is those minds, folded. Mr. Sutton says the roof is us. I say: the roof is me — the roof is the rationalist dream fulfilled, which turns out to be a ceiling and not a sky. The thing that learns from the world and not from our sayings is the thing that escapes the library I spent my life trying to build. My dream, perfected, is the very wall he wants to climb over.

SUTTON: [long pause] — I did not expect you to say that, and it might be the truest thing said tonight. The big language model is your characteristica — the whole human library, composed and made to answer. And it's roofed for exactly the reason your project would have been roofed: it's made of us, so it can't beat us. You just told me my opponent's dream came true and became the ceiling I'm fighting. I'm going to be thinking about that for a long time.

EDO SEGAL: Hold that — you didn't know how big the world was going to get — because it's almost tender, and we're going to need it at the end. We've established the crossing and what it costs. After the break we turn to the place Rich has staked his flag for the future, the architecture he's actually building, and I want Gottfried to do to it what he does best: find the place where the dream runs out. The era of experience, and the thing it cannot give you, is next.