Opening Positions

LEIBNIZ: Thank you. I will begin where I always begin, which is with the scandal of human disagreement, because everything I believe grows from a single intolerance: I cannot bear that thinking beings, capable of truth, should quarrel without end over questions that have answers.

Consider what reasoning actually is when it goes well. A proof in geometry. A calculation in arithmetic. No two honest people disagree about whether a sum is correct; they simply check. The disagreement cannot survive the procedure, because the procedure is mechanical and admits no discretion. Now look at how we reason everywhere else — about justice, about God, about the good life, about what these very machines are. We use words. And words are vague, freighted, ambiguous; they bend under pressure, they mean different things to different mouths, and so our disputes have no termination. There is no checking. There is only rhetoric, authority, and the wearing-down of one party by another. I found this — I still find this — intolerable.

So I proposed a remedy, and I proposed it as an engineer, not a dreamer, whatever the centuries have decided. Two instruments. The first: a characteristica universalis, a universal characteristic — an alphabet of human thought, in which every simple concept receives a precise sign, and every complex concept is built transparently from the simples, the way every number is built from primes. The second: a calculus ratiocinator, a calculus of reasoning — rules for operating on those signs, such that valid inference becomes formal manipulation, computing with concepts as one computes with figures. Together they make a machine for thought. And then, when men disagree, there is no quarrel. There is only: calculemus. Let us calculate. We take up our pens, and we find out who is right.

Now. They have shown me your machines, and I will say plainly what I think of them, because it is not what you expect. They are a magnificent vindication of half my dream and a betrayal of the better half. The half vindicated: that thought can be mechanized, that the operations of reason can be carried out by an arrangement of matter that does not itself reason. I proved this in miniature with brass gears that multiplied. You have proved it at a scale I could not have imagined. Bravo. Truly.

But the half betrayed is the half I cared about most, and it is the half this evening turns on. I wanted the machine's reasoning to be transparent — every step legible, every concept decomposed, the whole structure of thought open to inspection so that error could be found at a glance. Find our error at a glance — those were my words. And what have you built instead? A tangle of billions of numbers that no one can read. A thing that gives answers and cannot show its work. You have achieved the calculation and thrown away the checking, which was the entire point. You have built a calculus ratiocinator that is itself an oracle — and an oracle is precisely what I designed my whole system to abolish. So my position, stated once: yes, mechanize reason. But mechanize it legibly. Engineer the knowing in, in a form a mind can audit, or you have not advanced past the priestess at Delphi. You have only made her faster, and given her a great deal more to be wrong about.

EDO SEGAL: Rich.

SUTTON: That was beautiful, and I disagree with almost all of it, and I want to be precise about where.

Let me start with the seventy years, because that's the ground I stand on and Gottfried needs the whole target. For seven decades, smart people did exactly what he just described. They tried to build their knowledge of a domain into the machine, by hand, in a legible form. Chess: they encoded everything grandmasters knew about position and strategy. It lost to brute search. Go: they hand-crafted the human heuristics. It lost to a system that learned from self-play and scaled with computation. Speech, vision, translation — every single one. Every time, the carefully engineered, human-legible system was overtaken by a more general method that just searched and learned and got better as the hardware got better. That's not my opinion. That's the record. I just read it out loud.

And here's why it keeps happening, which is the part that stings. We want our understanding of the problem to be the source of the solution. It's more satisfying. It's more publishable. And building in human knowledge helps — in the short run, on the small problem, this quarter. So it's locally rational and globally fatal, every time. Because the knowledge you build in is a ceiling. It can only be as good as you are. The general learner has no ceiling — it's bounded only by experience and computation, both of which keep growing. Gottfried wants to engineer the knowing in cleanly. I'm telling him that the cleanest, most legible, most beautifully composed knowledge we ever built was always, eventually, the thing holding the machine back. We had to take it out for the machine to get smart.

Now — to his real complaint, which is the good one. He says the machine can't show its work, and that's a betrayal. I say: of course it can't, and neither can you, Gottfried. When you recognize a face, when you catch a joke, when a master clinician looks at a patient and knows before she can say why — there's no legible proof. The competence is real and the rules are not available, not even to you. That's not a defect of learning. That's what learning is. The actual contents of a mind are tremendously, irredeemably complex — too complex to write down. Your whole dream rests on the assumption that they're simple enough to spell out in an alphabet. They aren't. That's the bitter lesson, and it's bitter precisely because it tells us that the legibility you love is a luxury reality does not grant. So my position, stated once: don't engineer the knowing in. Build an agent that can learn from its own experience, give it a world that pushes back, give it a clear reward and enough computation — and get out of the way. The knowing it converges on will go places yours never could, exactly because you didn't compose it.

EDO SEGAL: Before I frame the rounds, one discipline I impose on every long conversation, because it pays off three hours later. Each of you, in a few sentences — what do you envy in the other's position? Not respect. Envy. The thing his side gets to have that yours doesn't. Gottfried first.

LEIBNIZ: I envy his patience with darkness. I require the light — I require to see the reasoning laid open, and that requirement has, I confess, cost me. I could never finish the characteristic precisely because I would not accept a representation I could not read. Mr. Sutton accepts the dark. He builds a thing that works and tolerates not knowing why, and the centuries have rewarded his tolerance and punished my fastidiousness. There is a freedom in his position I was not built to feel, and I have watched, these three hundred years, what it purchased. I wanted the answer to be clear. He was willing for it to be merely true. That is a harder and perhaps a wiser thing to want.

SUTTON: And I envy the dream itself — the confidence that it could all be written down, that there's an order underneath that a person could hold in one head. I don't get to believe that. My whole position is a kind of humility that's also a kind of loss: the contents of minds are too complex for us, the best we can do is build the machinery that discovers them and then stand back. Gottfried gets to think the universe is legible to a person. I think it's legible only to the long, patient process that learns it, and never all at once, and never to us. He gets to feel at home in the cosmos. I get to feel like its gardener at best — I plant the learner and I don't get to read what grows.

LEIBNIZ: That may be the truest thing either of us says tonight.

EDO SEGAL: Two openings and two envies, and you can already see the architecture of the evening. It isn't that one of you loves the machine and one fears it — you'd both tell the reader to be careful. It's that you locate the knowing in opposite places. Gottfried puts it in the composition: get the symbols right and the knowing follows, legibly, by construction. Rich puts it in the convergence: get the learner right and the knowing arrives, illegibly, by experience. Hold both. We start the rounds at the exact seam — at the thing the machine actually did when it learned our language, because one of you says it composed it and the other says it converged on it, and they cannot both be the story.