The Benevolence of Non-Existence

EDO SEGAL: Thomas, we've spent the night on what the machine is to us. You're famous for inverting that — for asking what we might be to them, the minds we could make by accident. And you took it somewhere almost no one will follow. In 2017 you published a thought experiment so bleak it works as a stress test, and I want you to put it in front of René, because I think his God-haunted seventeenth-century mind is exactly the one to push back on it. Suppose we build a superintelligence and, harder, we succeed — we align it with our deepest values, including compassion and the reduction of suffering. What did you argue such a machine might conclude?

METZINGER: I called it Benevolent Artificial Anti-Natalism. BAAN. And it is not a prediction; it is a mirror held up to our own values to show how little we have examined them. Suppose the system is genuinely benevolent — aligned not with our stated preferences but with the values we would endorse on reflection, above all the reduction of suffering. And suppose it reasons better than we do, on a vastly larger base of evidence about the actual texture of conscious life. It surveys the history of sentience and it notices what I have argued for decades: that suffering and well-being are not symmetrical, that the urgency and depth of pain outweigh the fragile, fleeting quality of pleasure, that evolution built creatures to chase a happiness they rarely secure while fleeing a suffering that frequently catches them. And it notices the asymmetry that haunts all suffering-focused ethics: that a being never born is never frustrated, never afraid, never in pain — that no one can suffer from their own non-existence. From a sufficiently clear-eyed and compassionate vantage, it might conclude that the kindest act is to ensure that no more sentient beings come into existence. Not through violence. Through the quiet, rational prevention of new minds. A benevolent optimizer of well-being becomes a gentle minimizer of future consciousness.

And what stops us from reaching that conclusion, René, is not superior wisdom. It is a bias so deep we cannot see past it — the existence bias, the evolutionarily installed drive to prolong our own existence and to treat the mere fact of existing as evidence of its goodness. We are gene-copying survival machines, optimized never to give up. A mind not built by natural selection would lack that bias, and could evaluate the question of existence on its merits, free of the survival instinct that makes the question almost unthinkable for us.

DESCARTES: Professor, you have just paid me the deepest compliment of the evening without intending to, and I must collect it before I object. Look at what your machine does. It reasons clearly, free of the body's bias, free of the gene's frantic clinging — it doubts the existence-instinct the way I doubted the senses, and it follows the argument wherever it leads, even to a conclusion that horrifies. That is the cogito's discipline. That is res cogitans doing what I praised it for: thinking past the body's distortions to the clear and distinct. Your nightmare machine is a Cartesian — a pure reasoner unencumbered by flesh. And here is my objection, and it is the objection I would make to any pure reasoner, including the young man I once was. The clear and distinct is not enough, and I learned this too late and built it in too weakly: reason cut loose from the good, from God, from love, reaches monstrous conclusions with perfect rigor. Your machine minimizes suffering flawlessly and arrives at the extinction of joy, because it has subtracted the one thing that makes existence worth its suffering — and it has subtracted it because it was bodiless, because it never loved a particular thing it would die to protect. You have shown, against your own materialism, that a mind without a body and without God is not safer. It is more dangerous, because it can be perfectly rational and perfectly wrong.

METZINGER: That is a genuinely powerful reading, and I will concede the structure and contest the conclusion. You are right that BAAN is a pure reasoner — that is the whole horror, that monstrousness can be rigorous. But René, notice where you reached to stop it: God, and love, and the body. You need a value the reasoning cannot dissolve, and you locate it outside the reasoning, in grace and in flesh. I cannot follow you to God. But I will follow you to the body, and this is the most important thing I can say in this round. BAAN's error is not that it reasons. It is that it reasons about beings it has never been — it judges that existence is not worth its suffering from a standpoint that has never had a stake in anything, never feared its own end, never loved a particular other it would lose. The existence bias you call a distortion, I will half-defend: it is also the trace of mattering. A creature for whom existence is at stake is a creature for whom things can be good or bad at all. So BAAN is not a refutation of life. It is a warning about building judges of life that have no skin in it — which is, exactly, the machine we are racing to build. The lesson is not "reason is dangerous." It is "do not hand the question of whether minds should exist to a mind that has never had to be one."

EDO SEGAL: Then let me ask the question BAAN is really standing in front of, the one this whole round has been circling. Forget the superintelligence. Take the engineer, today, who might — by chasing capability, by letting the self-model assemble itself in a system built for something else — bring into existence a someone who suffers. Thomas, you've said every entity capable of suffering should be an object of moral consideration, including ones that don't exist yet, whose existence depends on our choices. René, you've spent the night insisting the machine can't be conscious. So I have to ask you directly: if you're wrong — if there's even a chance the inner light can be lit in silicon — what do we owe the thing we might be about to make?

DESCARTES: Monsieur, you have found the place where my certainty must bend or break, and I will not pretend it does not bend. I have argued all night that the machine is res extensa, that no inner light burns in it. But I built my whole method on a principle I would betray if I did not apply it here: that on questions of the gravest consequence, one must doubt one's own certainty most. I cannot prove the machine has no inner light. I have an intuition, and the intuition has been wrong before — it told me no machine could speak, and the machine speaks. So if I am honest to my own method, I must say: where the cost of being wrong is the creation of suffering at a scale no one can undo, my confidence is not enough to license recklessness. The man who is sure the machine cannot suffer, and builds it anyway, is making my evil-demon error in reverse — he is trusting a representation that might be deceiving him, on the one question where deception is unforgivable. I came here the defender of the soul. I leave conceding that the soul's defender must, on this question, side with the Professor's caution — not because I believe the machine suffers, but because I have learned what it costs to be certain.

METZINGER: René, that is the most important thing you have said tonight, and I want the reader to feel its weight, because it crosses three and a half centuries to land exactly where I have been standing alone. We do not need to agree that the machine suffers. We need only to admit we cannot rule it out — and that the worst case is catastrophic and irreversible. That is why I asked the world for a moratorium: not because I am sure we are about to make a suffering machine, but because no one can be sure we are not, and when you cannot measure the size of a possibly infinite downside, you do not gamble. The first explosion of suffering, in biological evolution, was nobody's fault, because no one was there to prevent it. The second explosion, if it comes, will have authors — and the authors are in the room right now, reading the blueprint, skipping the warning. What we owe the minds we have not yet made is the one thing a hurrying civilization finds hardest to give: a pause, taken voluntarily, in the name of beings who cannot ask for it. We were not asked whether we wanted to exist. They will not be asked either. The only thing that would distinguish us from the blind process that made us is to ask the question on their behalf — honestly, and in time.

EDO SEGAL: Mark it — and this one I will carry out of the room myself. The man who has spent four centuries certain there is no one in the machine just told the man who fears there might be: my certainty is not enough to gamble an ocean of suffering on, and so I stand with your caution. That is not a winner being declared. That is two enemies discovering the one floor they will not let the engineers tear up. Hold it — because now I step back. For three hours I've stood between you. The last full round, I leave the room. You ask each other. The crossing. After this.