The Danger and the Saving Power

EDO SEGAL: Professor, you took a line from Hölderlin and put it at the center of your thinking about technology: "Where the danger is, grows the saving power also." Not optimism — you were clear it is not optimism. A structural claim: the saving power does not arrive from outside to rescue us from the danger; it grows within the danger, in the same soil, and can only be seen by those who first see the danger as danger. Ray, you are the optimist of the curve, but even you have an alignment problem you cannot wave away — the gap between what these systems can do and whether they are aimed at us. So this is the round where I want to find out whether your danger and his danger are the same danger. Ray, what keeps you up?

KURZWEIL: What keeps me up is the gap my own framework cannot close, and I have been honest about it for a long time, so I will be honest here. The Law of Accelerating Returns describes the trajectory of capability. It says nothing about the trajectory of wisdom, or alignment, or whether the power gets pointed at human flourishing. Capability research has the full force of the exponential behind it — each generation of models is built faster by the last. Alignment research does not get the same flywheel; understanding what a system is doing and ensuring it keeps doing what we want does not obviously get easier as the systems get more complex. So the gap between what the technology can do and what our institutions can govern is itself widening exponentially. My faith — and I will call it faith, not proof — is that the same exponential that opens the gap will give us the tools to close it. But that is a wager, not an argument, and the consequences of losing it are of a different order than any bet the species has placed before. That is what keeps me up: not that the curve is wrong, but that the curve is morally neutral, and it is accelerating into hands that may not be ready.

HEIDEGGER: Now listen — because that is the first thing he has said all night that I can stand beside, and I want to mark exactly where we meet and exactly where we part. We meet here: the danger is not the machine's malfunction. The danger is the machine working perfectly, in a frame that has forgotten to ask what the perfection is for. He calls it the gap between capability and wisdom. I call it the Ge-stell reaching completeness — everything, finally even the human, revealed as orderable resource, with no clearing left from which to ask whether the ordering serves anything. And here is where I must correct his hope, gently, because his hope is the most dangerous thing he owns. He says the exponential that opens the gap will give us the tools to close it. But you cannot close the gap with more of the thing that opened it. You cannot think your way out of the forgetting of Being with a better calculation, because calculation is the medium of the forgetting. The saving power is not a tool. It is not the next model. It is a change in our relation to the whole — and that change does not come from acceleration. It comes, if it comes, from learning to stand still inside the acceleration and ask the question the dashboard cannot generate.

EDO SEGAL: Then let me press you, Professor, because this is where your critics have always pressed you and they are right to. If the saving power is not a tool, if it comes from "standing still" and waiting for a turning that is an event of Being and not of human will — then what does the builder do tomorrow morning? The engineer in Trivandrum, the teacher, the parent. You can sound, forgive me, like a man telling people to wait for a god while the water rises. Ray builds. What do you tell the people who have to build?

HEIDEGGER: It is the right accusation and I will not dodge it, because dodging it has been the failure of my whole tradition. You are correct that "wait for the turning" can become a quietism that abandons the field at the moment it is most needed, and I will say plainly: that quietism is a temptation in my thought and it must be refused. So here is what the builder does tomorrow, and it is not nothing, and it is not waiting. She builds — and she maintains, inside the building, a clearing the system does not value. A space the dashboard cannot measure. The protected hour with the apprentice where the knowledge passes through the slow, inefficient, irreplaceable medium of one human attending to another. The refusal to let the machine do the attending even while it does the plumbing. The deliberate cultivation of an attentional ecology — a husbandry of the one resource the frame is strip-mining, which is the capacity for the meditative thought it cannot reward. These are not retreats from building. They are dams — built in the current, not to stop it but to keep alive, in the pool behind them, the thing the current would otherwise carry off. Mr. Kurzweil and I disagree about almost everything. We may not disagree about the dams.

KURZWEIL: We may not. I will go further than the professor expects — those dams are real and necessary, and I have argued for versions of them: the protected mentoring time, the deliberate friction, the spaces where humans build trust at the speed of trust rather than the speed of inference. Where I diverge is that I think they are bridge technologies — crude, leaky, temporary structures, like the four factory inspectors England assigned to police all its mills in 1833. Inadequate, and the beginning of something. The professor wants the clearing protected forever, as a permanent reservation against the frame. I want it protected for the transition — long enough for the merger to deepen to the point where the dichotomy he is defending, human-in-the-clearing versus machine-in-the-frame, stops being the relevant one, because the thing in the clearing and the thing in the frame have become one thing. He builds the dam to preserve the old river. I build it to survive the crossing to the new one.

HEIDEGGER: And that is the whole difference, said cleanly at last: he builds the dam to get across, and I build it because there is nothing on the far side worth crossing to if the clearing does not survive the trip. He treats the human-in-the-clearing as a stage to be transcended. I treat it as the thing itself — the candle that is not the lamp we are building toward but the only light there has ever been. If the merger he wants arrives and the clearing has not survived it, then the larger mind on the far shore will be the most capable thing the universe has ever produced and there will be no one in it for whom anything is at stake — a cosmos that computes everything and means nothing, the forgetting of Being made total and called the dawn. That is the danger. And the saving power, if it grows, grows in exactly the builders who keep the candle lit while they cross — who refuse, even amplified, even at the knee, to let the last clearing close.

EDO SEGAL: Mark this moment, because it is the closest the evening will come to a treaty, and notice what it took: it took the danger to produce it. The two men least likely to agree about what these machines are have agreed about what the builder must guard — a clearing, a candle, a dam in the current. They disagree, totally, about whether it is a permanent home or a temporary bridge. Hold that disagreement, because the last full round is about exactly that fork: the turning, or the sixth epoch. Whether what is coming saves us by changing our relation to Being, or saves us by waking the universe up. After this — and then I hand them to each other.