Let Us Calculate

EDO SEGAL: Gottfried, the boldest idea of your life was that one day all human disagreement could be ended by computation — that two people who differed would simply take up their pens and say to each other, let us calculate, the way two accountants resolve a discrepancy. It is the taproot of the whole field. And here is the irony I want you to sit in: the machine that finally reasons in something like the open-ended way you dreamed of did it by abandoning your method entirely. Not your universal characters, not your formal rules — but oceans of statistics over meaningless numbers. So tell me. Did your dream come true, or did something wearing its clothes come true instead?

LEIBNIZ: Both, and the distinction between them is the most painful and most instructive thing I have to learn from your century. My dream had two halves. The characteristica universalis — a language in which every concept is a character, complex ideas built from simple ones the way numbers are built from primes. And the calculus ratiocinator — rules for manipulating those characters so that valid inference becomes a kind of arithmetic. Put them together and disputes dissolve into calculation. Now: where the world could be formalized — in geometry, in logic, in the closed domains where concepts are crisp and rules exact — my dream came true with a vengeance. A theorem prover verifies a proof; a logic engine settles whether a conclusion follows. Calculemus is a daily reality there, and it is the foundation of your computer science. I was right that some reasoning is calculation. I was wrong about how much of human thought is.

MINSKY: And that's the part where I get to say, gently, welcome to my entire career, because I made the same beautiful mistake and it cost the field twenty years. We sat in that MIT lab and tried to do it your way, Gottfried — write the concepts down, the rules, the explicit symbols a machine could chain. I invented frames to hold the knowledge: a packet of default expectations for a stereotyped situation, a restaurant frame with its waiter and its menu and its bill, ready to fire on a cue. It was the right description of what a mind must have. It was a hopeless prescription for how to install it, because the world will not fit. For every obvious fact you write down, ten more obvious ones are presupposed by it. Try to write all the rules for what counts as a chair, or what happens when you pour water, and the project never terminates. We hit a wall, and the wall had a name — common sense — and it was made of millions of things too obvious to ever state.

EDO SEGAL: And then the wall came down. Tell the reader how, because it's the hinge of everything.

MINSKY: It came down by giving up on writing the knowledge by hand and letting the machine soak it up from the immense record of everything humans have written, where the obvious facts are encoded everywhere by implication. Ask a modern system what happens if you drop a glass, and it knows, not because someone told it but because the common sense precipitated out of the text. My frames survived — the restaurant comes complete with its waiter and its bill — but they were learned, not built. I had the destination dead right and the route dead wrong. And here's the part that should make Leibniz sit up: it didn't run on his clean symbols. It ran on the thing his whole tradition was built to avoid — continuous vectors, meaningless numbers, correlation instead of logic. The dream of mechanized reason came true, and it came true by betraying the dream's defining assumption: that thinking is calculation over clear symbols.

LEIBNIZ: Which is the deepest wound your century deals me, and I will not flinch from it. I bet that thought is, at bottom, calculation over distinct, meaningful, readable characters. The machines that actually reason in the open way are built on the opposite bet — competence from statistical pattern over representations no one can read. If you had asked me whether genuine reasoning could be built from a pile of meaningless numbers tuned by counting, I would have said no, with confidence, and I would have been wrong about what is possible. I concede it. But — and here I take a measure of revenge — notice what was lost in the betrayal, because it is exactly the thing my symbols provided. Stateable reasons. Verifiable steps. The capacity to show your work. Your vector machine answers and cannot tell you why; it confabulates with total fluency; it has no guarantee of consistency, no proof its conclusion follows. The field is already creeping back toward me, trying to wed the fluent network to a logical scaffold so it can not only answer but justify. My fork was not a dead end. It was one half of a circuit your century is still tracing.

MINSKY: I'll grant the revenge and sharpen it for you, because you're underselling it. The reason the field is crawling back toward symbols isn't nostalgia — it's that the statistical machine is brittle in precisely the way you'd predict. It knows that glasses break the way it knows everything: on average, by association, not by understanding why. So it fails strangely at the edges, confident and wrong, with no model of its own ignorance. That's the residue of the hardest problem, and it's the gap between a system that gets the right answer because it grasped the situation and one that gets it because the answer usually goes with the words. I spent my life wanting the first kind. The field built the second kind and called the war over. It isn't over. You and I, Gottfried — the man of symbols and the man of agents — we're on the same side of that one, against the people who think scale is a substitute for structure.

EDO SEGAL: Before I mark that, I want to press the dream itself, because there's a darkness in it I think you both see and I want it spoken. Gottfried, your calculemus was meant to be liberating — two people who disagree lay down their pens, calculate, and the truth settles the matter, no more bloodshed over what could be computed. But sit in our century for a moment. We now have machines that will, on demand, produce a fluent, confident, reasoned-sounding answer to any disputed question you put them — and people are already reaching for them exactly as you imagined, to settle arguments. Is that your dream coming true, or your nightmare? Because the machine says "let us calculate" and then hands you not a calculation but a plausibility, dressed as one.

LEIBNIZ: You have found the precise place where my dream curdles, and I will not pretend it does not frighten me. The whole point of calculemus was that the calculation would be checkable — that when two accountants disagree, a third can redo the sum and see who erred, because the steps are exposed and the rules are fixed. The authority came not from the machine but from the transparency of the procedure; anyone could verify it, which is why it could end a dispute without force. Your machine inverts this exactly. It produces the conclusion with all the confidence my calculation would have, and none of the exposed, checkable steps that were the entire source of the calculation's authority. It says "let us calculate" and shows you no calculation. That is not my dream fulfilled. That is my dream's costume worn by my nightmare — the appearance of settled reason, with the checkability that made reason trustworthy stripped out. A man who accepts the machine's answer because it sounds like a calculation has surrendered the one protection calculemus was invented to give him: the right to redo the sum himself.

MINSKY: And here's the cruel part I'd add, because it's worse than he's saying. The machine doesn't just hide the steps — it produces reasons on demand for whatever conclusion you lean toward, because it learned from a corpus full of humans arguing every side of everything. Ask it to defend a claim and it will; ask it to demolish the same claim and it will, just as fluently. So it can't settle a dispute the way Leibniz wanted, because it has no stake in the truth — it has a stake in plausible continuation, and plausibility is available on both sides of almost any real disagreement. The dream of mechanized reason assumed the mechanism would be anchored — to logic, to proof, to something that makes one answer right and the other wrong. The machine we built is anchored to nothing but the shape of human talk, which contains every answer. It's a calculemus engine with the truth-anchor removed, and people are using it as if the anchor were still attached. That's not a small bug. That's the thing that turns the dream into a flood of confident noise.

EDO SEGAL: Mark this — convergence number one, and I'm numbering them tonight because when these two agree it's news. Both of you, from opposite centuries and opposite methods, hold that fluency without stateable structure is a kind of poverty, not a kind of mind — that the machine which answers without being able to justify has shown you the wake, not the boat. Marvin, you'd add it's still a mind; Gottfried, you'd say that's exactly why it isn't one. But you agree the missing thing is real. Hold that agreement — it's load-bearing in the round on reasons. Because Marvin made a move three minutes ago I won't let pass: he keeps saying "consciousness," "understand," "self" are suitcases, words packed with six unrelated things. Next round, we open the suitcases. And Gottfried, I suspect you think the suitcase trick is how Marvin avoids the one bag he can't unpack.