The Question on the Table

EDO SEGAL: Somewhere in the world right now, in the time it takes me to say this sentence, a few hundred thousand people are typing a disagreement into a machine and asking it to resolve them. A couple arguing about a medical bill. A manager and an engineer who cannot agree on what the data shows. A teenager and her own doubt, at midnight, asking the box whether the thing she believes about herself is true. And the box answers — fluently, instantly, with the calm authority of something that has read everything and forgotten nothing. It does what no priest, no judge, no parent could ever quite do. It says: here is the answer. Stop arguing. I have calculated.

One of the two men at this table dreamed exactly that machine three hundred years before it existed, and called its arrival the end of human discord. The other built it, more or less, and then walked away from the most powerful institutions in the field to warn us that a machine which calculates perfectly is the most dangerous thing we will ever make. So here is the question we are going to spend three hours inside, and I'll state it once, plainly, because every round tonight is this question wearing a different coat: if a mind could compute the perfect answer to everything, would you trust it with the meaning of your life — or is the wanting itself the part no calculation can hold?

I have wanted this conversation for a very long time, and I will be honest that arranging it required bending the rules of time and death, which I'll address in a moment. Let me introduce my guests.

Gottfried Wilhelm Leibniz was born in Leipzig in 1646 and was, by wide agreement, the last human being to hold the whole of European learning in a single head. He invented the calculus — independently of Newton, and published it first. He devised the binary arithmetic of zeros and ones on which every machine in this building now runs. He built a calculating engine of brass and steel. And he proposed, in detail, something he called the characteristica universalis: a universal language of thought, paired with a calculus of reasoning, in which any dispute — moral, legal, metaphysical — could be settled by computation. He gave us the phrase that is the closest thing this whole field has to scripture: calculemus. Let us calculate.

LEIBNIZ: You are generous, and only slightly inaccurate, which from a host is the most one can hope. I would only correct the word built. I designed an engine that multiplied and divided. Whether it always multiplied and divided correctly is a matter on which my craftsmen and I did not always agree. The gears, monsieur, were less obedient than the idea. That distinction — between the idea that is perfect and the brass that betrays it — may turn out to matter tonight.

EDO SEGAL: It will matter enormously. Nick Bostrom needs less introduction only because he is of our own century. A Swedish-born philosopher, trained across physics, computational neuroscience, mathematical logic, and philosophy — which is to say, trained to reason about minds as physical systems and then to ask what such systems would mean. In 2005 he founded the Future of Humanity Institute at Oxford and built it into the intellectual home of a question almost no serious academic would touch: what happens to a species when it builds a mind greater than its own. His 2014 book Superintelligence moved that question from the fringe of futurism to the center of global strategy. He gave us the vocabulary we now live inside — the orthogonality thesis, instrumental convergence, the treacherous turn, the vulnerable world, deep utopia. We are, as I've written, living inside his glossary.

BOSTROM: That's roughly accurate, and I'll accept it with one caveat that I suspect will also matter tonight. I'm not a prophet of doom, and I'd resist the framing if it crept in. I reason in the conditional. I don't claim to know what will happen. I claim that if certain things happen, certain consequences follow with high probability, and that some of those consequences are irreversible — and that irreversibility is what changes the math. I'm here to be precise, not to be frightened, though I grant the two can be hard to tell apart.

EDO SEGAL: And there's the seam already. Before we go further, I owe the reader an honesty about the strangeness of this room, because one of my guests has been dead for three hundred and ten years. Gottfried — you did not live to see electricity, let alone a transformer. For the purpose of this evening, you have been briefed. You know what these machines are, how they're built, what they can do and where they fail. You're reacting to the present, not predicting it. I want to acknowledge that once, lightly, and then never insult either of you with it again.

LEIBNIZ: I find the condition agreeable, and less disorienting than you might fear. I have read your accounts of these systems with the particular vertigo of a man reading his own unfinished correspondence completed by another hand. You built the calculus ratiocinator. You built it out of probability rather than logic, in a notation no one can read, by a method I would have called barbarous — and it works, after a fashion, which is more than my version ever did. I am not here to be a relic. I am here because I asked the first version of every question you are about to ask, and I should like to see how my answers have aged.

BOSTROM: They've aged in an interesting way. Some have aged into vindication and some into warning, and I think the warnings are the ones that came from him, not at him. That's part of why I wanted to sit across from him specifically.

EDO SEGAL: Then let me set the rules of the evening — there are three, and you may each add one. First: we have three hours, which means no one has to win by the next bell. The whole point of long form is that you can let an argument breathe before you strangle it. Second: I declare my bias at the door. I build with these systems daily. I wrote a book with one. I have skin in this question on both sides of my own heart — I am moved by what Leibniz hopes and chilled by what Bostrom fears, and I will tell you, out loud, when my stake is touched. Third: at the end, no one wins. If the disagreement survives three hours, we hand it to the reader, intact. Gottfried, a rule of your own?

LEIBNIZ: Then mine is the one I have always insisted on. We must distinguish the calculation from the characteristic — the operation from the notation it operates upon. When this gentleman says the machine optimizes, I will want to know: optimizes what, expressed in what signs, and who fixed the signs. Half of every dispute I ever witnessed was not a dispute about reasoning at all. It was two people using one word for two things. I will be insufferable about definitions. I was in life. I see no reason for death to have improved me.

BOSTROM: Then mine is the symmetric one. When Leibniz says reason converges on the good — that a sufficiently clear mind cannot help but be a benevolent one — I'm going to ask him to cash it out as a mechanism, not an intuition. Name the step in the calculation where goodness gets in. Because my entire position is that there is no such step, that you can run the calculation to infinite precision and the goodness simply never appears unless you put it in by hand. If he can show me the step, I'll change my mind on the spot. That's the rule: no appeals to what a wise being would surely see. Show me where, in the math, surely lives.

EDO SEGAL: You see why I wanted this. Two rules, and they're already the same disagreement: Leibniz says the good is in the calculation if you do it cleanly; Bostrom says the good has to be loaded into the calculation from outside, and that nothing about doing it cleanly will supply it. Hold that. Because here is the question on the table, stated once more, plainly: if a mind could compute the perfect answer to everything, would you trust it with the meaning of your life — or is the wanting the part no calculation can hold? Gottfried Leibniz, you have waited three centuries. The floor is yours.