The Relentless Arithmetic and the Confident Wrongness

EDO SEGAL: Hans, your timeline — the mind children arriving, human-level machines in the 2030s and 40s — none of it comes from philosophy. It comes from a curve. Lay out the arithmetic, because it's the load-bearing wall of the optimism, and then Aristotle, I want you on the question of whether a graph can tell you what's coming.

MORAVEC: It's the most disciplined part of my work and the part I'm least embarrassed by. The cost of computation has fallen exponentially for the entire history of the technology — through relays, vacuum tubes, transistors, integrated circuits, one substrate exhausting itself and the next picking up the same curve without missing a beat. I tried to estimate, in honest engineering units, how much raw computation a human brain performs — counting neurons, synapses, firing rates — and then asked when silicon that cheap would arrive. The numbers were rough; people argued my brain estimate by orders of magnitude. But the shape was robust. Whatever the exact figure, the relentless fall in the price of computing meant a machine with brain-equivalent power becomes affordable at a foreseeable moment, and after that the machines keep going and we don't. The whole modern era — the scaling laws, the bet that pouring in more compute reliably buys more capability — that's my wager rendered as a science. They got smarter mostly because they got bigger. I said that before it could be shown.

ARISTOTLE: And on that narrow point I will not fight him, because I respect a man who counts. But I want to expose the leap inside the curve, and it is the same leap as the upload, wearing different clothes. Moravec has measured the growth of processing and concluded the growth of mind. He has charted how fast the machine can compute and inferred how soon it will understand. But those are two different things, and his own paradox should have warned him: the machine got the easy veneer fast and the hard root not at all. A graph of operations-per-second is not a graph of comprehension. He has confused episteme of the fact — the machine knowing that the patterns fall thus — with episteme of the reasoned fact, knowing why. The machine produces fluent, confident prose that has the shape of insight and is hollow at the center. I have a name for the danger: it is confident wrongness dressed in good prose. The machine states what is false with the exact cadence of what is true, and your curve, Moravec, predicts none of that, because your curve measures the wax and not the seal.

MORAVEC: Now wait — you can't have it both ways, Aristotle. A minute ago embodiment was the deep real thing and reasoning was the thin veneer. Now you're complaining the machine is bad at reasoning — confidently wrong, hollow. Which is it? Is reasoning the cheap easy part the machine aces, or the hard part it fumbles? You're using my paradox as a club in both directions.

ARISTOTLE: A fair catch, and the answer dissolves the contradiction. The machine is excellent at the manipulation of symbols — the moves, the syntax, the surface form of reasoning. That is the cheap veneer, and it aces it. But reasoning in the full sense is not symbol-manipulation; it is the grasp of causes, the seeing of why, and that grasp is rooted in the same embodied engagement with a resisting world that the toddler has and the machine lacks. So the machine has the shadow of reasoning — the part that looks like reasoning from outside — and lacks the substance — the part that knows why. There is no contradiction: it has the form-of-the-form, the impression in the wax, and lacks the living understanding that cast it. That is one phenomenon, seen twice.

EDO SEGAL: Let me bring this down to a kitchen table, because my mother would lose us both right now. Picture a parent helping a twelve-year-old with homework, and the machine on the screen answers faster and cleaner than either of them. Hans, your curve says: give it a few more doublings and that machine simply knows more than the parent, full stop, and we should plan our lives around its arrival as a genuine mind. Aristotle, you're saying: that same machine, however fluent, may be confidently and invisibly wrong, and the parent's job — the irreplaceable human job — is to be the one who knows why, who can catch the hollow answer. For the parent at that table tonight: which of you is describing the world she actually lives in?

MORAVEC: Both, for now — and that's the honest answer, not a dodge. Today she lives in Aristotle's world: the machine is fluent and often hollow, and her judgment is the safeguard. I'm telling her about her grandchildren's world, where the doublings have continued and the hollowness has been filled in, because there's no law of nature that says the gap between fluent and grounded can't be closed by scale. She should trust her judgment today. She should not bet her grandchildren's future on the gap staying open forever.

ARISTOTLE: And I would tell her the opposite, and let her choose between us, which is the format. The gap between fluent and grounded is not a quantity that more computation closes. It is a difference in kind. You can double the impressions in the wax forever and never get a seal; you can sharpen the portrait to any resolution and never get the man. Moravec keeps promising that enough more of the same crosses a line into something else. That is the one thing quantity cannot do. A longer ladder does not reach the moon. He has built a magnificent ladder and mistaken its length for a rocket.

EDO SEGAL: "A longer ladder does not reach the moon" — Hans, that's the whole bear hug of the disagreement, and I want to give you the last word of the round, because he just said your life's central bet is a category error.

MORAVEC: And I'll say what I've said for fifty years against a field that was certain I was wrong: they told me learning couldn't scale, that pattern without rules was a parlor trick, that a network would never see. I bet on more-of-the-same crossing into something else, and I won that bet, against exactly this kind of "difference in kind" argument. Aristotle's "ladder to the moon" is what they said about every threshold the machines have already crossed. Maybe this one's different. Maybe the self really is the one thing scale can't reach. But I've stood in the room where the "impossible in kind" became "Tuesday," more than once, and it makes me slow to bet against the ladder.

ARISTOTLE: Then we have found the honest shape of it, and I will not pretend my certainty is his experience. He has watched walls fall. I have watched what walls are. The reader must decide whether the self is a wall that falls or the one that doesn't.

EDO SEGAL: Hold the ladder and the moon. We've now mapped the two great engines — the paradox and the arithmetic — and found a plank inside each that one of you leapt and the other won't let you forget. The next hour goes to the thing underneath all of it, the thing that makes this debate matter more than any other in the series: not whether the machine thinks, but whether anyone is home — in the machine, and in the copy of you that might wake inside it.