The Heuristics of Fear and the Wisdom Race

**EDO SEGAL:** I want to open this round with my own ledger, because the moderator should pay the table's toll first. For thirty years I built engagement machinery — products designed to capture attention, and I knew the loops, I knew exactly which human weaknesses I was renting. I told myself I didn't know the harm. I knew. And the thing I've learned, too late to undo it, is that the absence of proof of harm is not the same as proof of safety — that "we have no evidence it's hurting them" was the sentence I hid behind while it hurt them. So when I read both of you on how to act under uncertainty, I read with a guilty man's attention. Professor Jonas, you gave the world the *heuristics of fear*. Max, you gave it the *wisdom race*. They sound like allies. I suspect they're enemies. Professor Jonas, start.

**JONAS:** They are enemies, and you are right to suspect it, though Dr. Tegmark and I will surprise you by agreeing on the surface before we part at the root.

The heuristics of fear is not timidity, and it is not pessimism, and I have spent too long correcting both misreadings to let them stand. It is this. Fear is an *organ of perception*. The organism that evolved to detect threats with speed and precision — because the one that missed a threat died, while the one that missed an opportunity merely missed a meal — that organism *sees danger more clearly than it sees benefit.* This is not a flaw to be corrected. It is a faculty to be *used*. In conditions of genuine uncertainty about an irreversible action, the worse prognosis must be given methodological priority — not because it is more likely, but because the consequences of being wrong about it are categorically different. If I am cautious and the technology was benign, I have lost some time, and time of that kind is recoverable. If I am bold and the technology was ruinous, the ruin may not be recoverable at all. The asymmetry is the whole of the ethics. The burden of proof falls — and here I invert the entire culture of building — *on the one who claims the new power is safe*, not on the one who fears it.

Now apply it to *this*. Dr. Tegmark and I both think the stakes are enormous; we'll agree on the magnitude in a moment. But watch the *shape* of his urgency and the shape of mine. His says: the wisdom race is being lost, so we must *go faster on wisdom* — build the alignment, build the governance, run harder to keep up. Mine says: when the action is irreversible and you cannot foresee its end, the wisdom is not in *running faster*. It is in *refraining* — in the deliberate, informed decision *not to do what you are able to do*, because you cannot undo it if you are wrong. He wants to win a race. I want to question whether one should be running at all toward a cliff one cannot see the bottom of. That is the difference between a heuristics of fear and a wisdom race: one of them is willing to *stop*.

**TEGMARK:** This is where Professor Jonas and I are closer than the room thinks and then split hard, and I want to honor the closeness first because it's real. I *also* invert the burden of proof. I've spent a decade arguing that you shouldn't deploy a system whose [existential risks](https://www.youonai.ai/fieldguide/med/existential_risk) you can't bound — that the AI industry is like the pharmaceutical industry before the FDA, marketing drugs without proving safety, and that we need a body with the authority to say "demonstrate it's safe *first*." I signed the letter calling for a pause. I helped write a statement calling for a conditional prohibition on superintelligence until there's scientific consensus it can be done safely. So when Jonas says "the burden falls on the one who claims it's safe" — Professor, I've been shouting that into the wind for ten years. The precautionary instinct, I share completely.

Here's the split, and it's everything. Jonas's heuristics of fear, taken to its conclusion, has *no upside term in the equation.* It looks only at the downside of acting and says: when the downside is irreversible, refrain. But there's an irreversibility on the *other* side too, and his framework is structurally blind to it. Look at the [cosmic endowment](https://www.youonai.ai/fieldguide/med/cosmic_endowment). The observable universe holds something like 10^80 atoms; conscious life, as far as we know, exists on exactly one planet, a vanishing speck. If we navigate this transition well, intelligence and consciousness could spread through the cosmos for billions of years — an almost unimaginable flourishing. If we *fail to act* — if we refrain ourselves into stagnation, or let someone reckless get there first — that future is *also* lost, permanently, and the loss is just as irreversible as any catastrophe Jonas fears. The wisdom race isn't "go fast." It's: the power is accelerating whether we like it or not, so the only variable we control is whether wisdom accelerates *too*. Refusing to run doesn't stop the race. It just means the least wise runner wins. That's the part the heuristics of fear can't see — that *not building* is also an action, with its own irreversible shadow.

**EDO SEGAL:** Let me restate that collision as sharply as I can, because it's the practical hinge of the whole night. Professor Jonas says: facing an irreversible unknown, the wise act is to *refrain* — the burden is on the builder to prove safety, and if he can't, he stops. Max says: refraining is *also* an irreversible act, because the cosmic future is lost if we freeze, and anyway the race runs without you, so the only real choice is whether wisdom keeps pace. Professor Jonas — answer the cosmic endowment. He's saying your caution could cost the universe its future.

**JONAS:** He is, and I will answer it without flinching, because it is the most seductive argument in his arsenal and the most dangerous. Dr. Tegmark places on one side of the scale a *concrete, present, irreplaceable thing* — the child whose development is being reshaped now, the human capacity for genuine thought that is being eroded now, the actual living world. And on the other side he places a *speculative, vast, future abstraction* — 10^80 atoms, billions of years, the flourishing of consciousness across the cosmos. And he asks me to discount the concrete present against the speculative future, because the future is so *large*.

This is the oldest and bloodiest move in the history of dangerous idealism. *We must sacrifice the actual people for the magnificent future.* I have seen where it leads. The men who murdered my mother were also building a thousand-year future, also weighing present lives against a glorious abstraction and finding the present lives light. I do not say Dr. Tegmark is those men — he is a decent man and I believe him decent. I say the *structure of the argument* is the structure that has justified every atrocity of the technological age: the present, particular, breathing human is made small by comparison to the Future, and once you have made him small you may do anything to him. The heuristics of fear is precisely the refusal to be hypnotized by the large number. It says: the irreversible harm to the *actual* — the child, the body, the living condition we can see — outweighs the speculative loss of the *possible*, because the possible can take care of itself if the actual survives, and the actual cannot be restored if we gamble it away on a cosmos that does not yet, and may never, exist. You ask me to risk the candle that is burning for the trillion candles that might someday be lit. I will not. The burning one is the only one that is real.

**TEGMARK:** *[after a long pause]* I have to sit with that, because the comparison you just drew is not one I can wave off, and I won't insult you by trying. Let me say only this, and then I think Edo should move us, because we've found bedrock. You're right that "sacrifice the present for the magnificent future" is the most dangerous argument in history. Where I'll stand my ground is that I'm not asking to sacrifice the present child. I'm asking *for* her — because the same misaligned system that could foreclose the cosmic future is the one that could harm her next year, and the wisdom that protects her is the wisdom that protects them both. But I hear the warning under your warning. I think you've earned it the hard way, and I'd be a fool not to carry it out of this room.

**EDO SEGAL:** *[quietly]* Mark this — it's the first convergence of the night, and it's a strange one. You both invert the burden of proof. You both want the builder to prove safety before he ships. You both refuse the comfortable middle. *[a beat]* And then you split on a single question: whether the future is large enough to outweigh the present, or whether the present is real enough to outweigh the future. That's not a disagreement about AI. That's the oldest disagreement there is. Hold it. Because everything so far has assumed we even know whether there's *anyone home* in the machine to do any of this caring. Next round, the deepest water in the building. Is anyone home? After this.