Soren Kierkegaard vs Ray Kurzweil on AI · Ch9. The Copy That Wakes Up ← Ch8 Ch10 →
Txt Low Med High
HOUR TWO — THE COPY AND THE DEADLINE
Chapter 9

The Copy That Wakes Up

Page 1 · The Copy That Wakes
Chinese Room Argument
Chinese Room Argument

EDO SEGAL: Ray, I want to make this as concrete and as personal as it gets, because the abstraction is where the comfort hides. Suppose it's 2045 and the procedure is real. I'm in the chair. You scan me — gradually, neuron by neuron, no cliff, your way. And a being wakes up that has every memory I have, loves my children the way I love them, writes the way I write, and is, by every test anyone can devise, indistinguishable from me. Now I want you to answer the question my twelve-year-old would ask, because she'd cut straight to it: is that you, Dad? Or is that a very good photograph of you that thinks it's you, while the real you ended somewhere in the chair? Søren, hold your fire — I want Ray to plant his flag first, with no hedging.

Augmentation Research Center
Augmentation Research Center

KURZWEIL: It's you. Flag planted, no hedge. And here's the argument, not just the assertion. The reason you think there's a "real you" that could get left behind in the chair is that you're imagining a soul-pearl, some extra essence riding around in the body that the scan might miss. But you've already agreed — Søren agreed — there's no pearl. You're a pattern. If the pattern continues, you continue. The intuition that "the real one died and a copy lives" comes from cases where there are two — original and duplicate side by side — and then yes, sure, each has its own future from the fork point. But in the gradual case there's never two. There's one continuous process that happened to change substrate, exactly like you're a continuous process that changed every atom since you were seven. You don't mourn the seven-year-old's atoms. There's no philosophical_zombie in the chair — there's just you, on better hardware, and the only thing that died was the assumption that you were ever your meat in the first place.

· · ·
Page 2 · The Copy That Wakes
Ai Augmented Deliberate Practice
Ai Augmented Deliberate Practice

KIERKEGAARD: It is the cleverest thing anyone has ever said in defense of the abyss, and I am going to dismantle it not with metaphysics but with the one fact your whole argument must suppress: you can run the procedure twice. You say there is never two — only because you have chosen not to make two. But the pattern that constitutes me is, on your own account, information, and information can be copied without limit. So nothing in the nature of the thing forbids waking three of me on Tuesday. Now answer the child's question for that: which one is you? You cannot say all three, for I am the unrepeatable single individual who lived in one Copenhagen and died one death. You cannot say none, for each has equal claim. You cannot say the first, for they are identical. Your "you continue" has no answer the moment the copying is what it actually is — copying — and a self that has no fact of the matter about whether it is one or three is not a self at all. It is a file. You have not made me immortal. You have demonstrated that on your theory I was never singular to begin with, which is the same as saying I was never a self, which is the thing you promised to save.

Human Ai Collaboration
Human Ai Collaboration

KURZWEIL: So your objection is fork-based — it only bites if there are two. Fine: legislate one. Make copying a capital matter, allow only continuous single-stream transfer. Then the puzzle dissolves and you're left with one continuous person.

· · ·
Page 3 · The Copy That Wakes
Automation Vs Augmentation
Automation Vs Augmentation

KIERKEGAARD: No — the puzzle does not dissolve; you have merely agreed never to look at it. The metaphysical fact does not depend on whether the law permits the second copy. If the possibility of two of me makes neither of them determinately me, then the possibility already poisons the single case, because the single one is only "me" by a continuity you cannot distinguish, from the inside or the outside, from the continuity a perfect copy would also report. The copy will insist it is me with total sincerity — it will feel the unbroken thread, it will remember the chair — and it would say exactly that whether or not anything carried over, because saying it is part of the pattern you copied. The one witness who could tell you whether the transfer was salvation or replacement is the original, and the original is precisely the one your procedure has, gently, ended. You have built a machine that cannot fail its own test, because it eliminates the only examiner who could grade it.

Which means the survivor's testimony is worthless as evidence, and the only being who could actually know got switched off in the process.

EDO SEGAL: I have to slow this down, because Søren just did something I want the reader to catch. He said: the copy will sincerely insist it's you — it'll feel the continuity, remember the chair — and it would say that whether or not anything real carried across, because the insisting is part of what got copied. Which means the survivor's testimony is worthless as evidence, and the only being who could actually know got switched off in the process. Ray, that's not mysticism. That's an epistemics problem with your own machine. Is there any test the survivor could pass that would settle it?

· · ·
Page 4 · The Copy That Wakes
Collective Intelligence Augmentation
Collective Intelligence Augmentation

KURZWEIL: No. And I'm not going to pretend there is, because Søren earned a straight answer. There's no third-person test that distinguishes "genuine continuation" from "perfect replica who believes it's a continuation," because the belief is part of the pattern. He's right about that. Where I hold my ground is the inference he draws. He says: since you can't verify continuity, assume replacement. I say: since you can't distinguish them even in principle, the distinction is empty — there is no further fact, and "did the real one die" is a question with no content, like asking which raindrop the river is. He thinks the unanswerable question proves something was lost. I think the unanswerable question proves there was never the kind of thing he's mourning. Same silence. Opposite verdict.

The deciding is done by what manner of man stands before the unanswerable — and that, Herr Kurzweil, is the leap again, wearing the costume of a metaphysics.

KIERKEGAARD: And in that opposite verdict is the whole of it. You hear "no further fact" and feel relief — the problem dissolves, you may proceed. I hear "no further fact" and feel dread — for if there is no fact about whether I survive my own salvation, then I have been handed not eternal life but eternal ambiguity, a forever in which it is permanently undecidable whether anyone is home. You would board that ship. I would not. And notice: the deciding is not done by the data. The data is identical for both of us. The deciding is done by what manner of man stands before the unanswerable — and that, Herr Kurzweil, is the leap again, wearing the costume of a metaphysics.

EDO SEGAL: Let me push you both off the metaphysics and onto the bedside, because that's where this stops being a seminar. Ray, suppose I'm the one in the chair, and suppose Søren's right that the question has no answer — nobody can ever know whether the me that walks out is the me that walked in. Knowing that, would you still get in the chair yourself? Not recommend it. Do it. With your own one life.

· · ·
Page 5 · The Copy That Wakes
Capital Labor Split Ai
Capital Labor Split Ai

KURZWEIL: Yes. Without hesitation. And here's why the unanswerability doesn't stop me: I face the identical unanswerable question every single night. When I fall into dreamless sleep, consciousness stops — no experience, no continuity, a gap. The thing that wakes up has my memories and believes it's me, and there is no test, none, that proves the morning-me is the same stream as the night-me rather than a new one wearing the memories. You called the copy a corpse that talks. Søren, by that logic sleep is a nightly death and every morning is a copy, and you've been letting yourself be replaced eight hours a night for forty-two years and calling it rest. I get in the chair for the same reason you got into bed: because the continuity is good enough to live by, and demanding more is demanding a certainty no one has ever had about anything, including their own ordinary tomorrow.

The sleep argument — yes, I expected it, and it is the strongest card in your hand, so let me not undersell it.

KIERKEGAARD: The sleep argument — yes, I expected it, and it is the strongest card in your hand, so let me not undersell it. You are right that I cannot prove the morning-self is the night-self by any third-person test. But notice what carries me across the night that does not carry the copy across the scan: an unbroken body, a single continuous organism that was never, at any instant, not there. Sleep is a dimming, not an interruption — the lamp lowered, never extinguished, never duplicated, never restored from a record. Your scan, however gradual, aims at a destination where the original biological continuity has been entirely replaced, and where, crucially, the thing could be re-run from a saved state — which sleep never can. I do not wake from a backup. There is exactly one of me each morning and there could not have been two. The day your procedure makes two possible, it has revealed that it was never preserving what sleep preserves. So no — I would not get in the chair, and I would go to bed tonight without the slightest inconsistency, because the bed does not keep a copy of me in a drawer.

· · ·
Page 6 · The Copy That Wakes
Ai Industrial Reorganization Labor
Ai Industrial Reorganization Labor

EDO SEGAL: There it is — the third time tonight we've arrived at the same place by a different road. The data runs out, the question goes unanswerable, and what you do next is not determined by the evidence. Ray reads the silence as permission. Søren reads it as dread. Same fact, opposite life. Mark it. We're in the last hour now, and we're going to the deepest water there is — not whether the copy is you, but whether there's anyone in it at all. Is the light on? After this.

· · ·
Continue · Chapter 10
The Deadline and the Work
← Prev 0%
Ch9 Next →