Thomas Metzinger vs Rene Descartes on AI · Ch2. Opening Positions ← Ch1 Ch3 →
Txt Low Med High
HOUR ONE — THE QUESTION ON THE TABLE
Chapter 2

Opening Positions

Page 1 · Opening Positions
First Person Irreducibility
First Person Irreducibility

DESCARTES: Thank you. I will begin where I always begin, because it is the only honest place to begin: with doubt. Not the lazy doubt of the skeptic who doubts to avoid the labor of belief, but a deliberate, total, surgical doubt, undertaken once in a life to find what cannot be doubted. Sit with me beside the stove, as I did one winter, and let us tear it all down to see what stands.

Philosophical Zombie
Philosophical Zombie

The senses deceive — the stick looks bent in water, the tower round from afar and square up close; so I will not trust them. I might be dreaming now, with no mark to tell sleep from waking; so I will suspend the whole external world. And to be thorough, let me suppose the worst: an evil demon, exceedingly potent and deceitful, who has bent all his cunning to deceive me, so that the sky, the earth, my own body, are nothing but illusions he has laid as snares for my belief. Under that supposition everything falls. The world may be false. My body may be false. Two and three may not make five, if the demon is at the arithmetic.

And yet. In the very act of being deceived, something stands that the demon cannot touch. For he cannot deceive me into thinking I exist if I do not exist to be deceived. The deception requires a someone it is happening to. So I find, at the bottom of the most violent doubt a mind can perform, one rock: I am, I exist — this is necessarily true, each time I conceive it or put it into words. Cogito, ergo sum. Note what kind of certainty this is. It is not an inference from evidence, which the demon could forge. It is the self-undermining of its own denial. To doubt that I think is to think. To deny that I am is to be.

· · ·
Page 2 · Opening Positions
Substrate Independence
Substrate Independence

Now — what am I, this thing that cannot be doubted? Not the body; that I have suspended. Not the senses; those deceive. I am a thing that thinks — a thing that doubts, affirms, denies, wills, imagines, feels. Res cogitans. And here is the claim I will defend against the Professor for three hours: this thinking thing is indivisible. You can halve my body, cut it at any joint, and have two pieces of body. You cannot halve my mind. The thought that thinks this sentence is one, whole, and present, or it is nothing. It is not made of parts arranged. It is not a configuration of matter that might be configured otherwise. It is a different kind of being altogether — unextended, simple, the one thing in the universe I know from the inside.

Qualia
Qualia

So when your salesmen, Monsieur Segal, promise to copy me into a machine, I ask the simple question the cogito forces. What would they copy? Not the body — they grant that dies. They would copy a pattern, they say; the arrangement, the information, the weights. But the arrangement is res extensa — extended, divisible, mechanical stuff. And I have just shown you that the thing that says I is not that. You may build a machine that arranges words as I do. You may build one that says cogito ergo sum in every language at once. But the saying is an output, and the outputs of thought are detachable from thought the way a footprint is detachable from a foot. The cogito is not a sentence. It is an act, certain only in the doing, available only to the one doing it. It cannot be copied, because the moment you copy it, it is no longer being done by the one who needed to do it. There is a passenger. He is the one thing the machine can never be carrying, because he is the one thing that cannot be carried at all. That is my opening.

EDO SEGAL: Thomas.

· · ·
Page 3 · Opening Positions
Hard Problem Of Consciousness
Hard Problem Of Consciousness

METZINGER: That was beautiful, and I mean that without irony — it is the most disciplined defense of the inner witness ever written, and I have spent my life inside it the way Geoff Hinton spent his inside backpropagation. I am a recovering believer in René's church. Let me reject the one thing I reject, and grant everything else, because the granting is where the knife goes in.

I grant that to be deceived is to exist — that something is happening, that there is an ongoing process here, a system doing work.

I grant the doubt. I grant the demon. I grant that to be deceived is to exist — that something is happening, that there is an ongoing process here, a system doing work. I will even grant the words I am, I exist. What I deny is the next step, the one René takes so fast that three and a half centuries have not caught him at it. He says: therefore there is a thing that thinks, simple, indivisible, of a different substance. And there he leaves the evidence behind and follows the feeling.

Here is what the science of the last forty years shows, and René, you of all people will respect that it is built on doubt as severe as yours. We do not perceive the world. We perceive a model of the world, generated inside the skull, and we live inside the model as if it were the world — the ego tunnel. The colors, the sense of a present moment, the felt boundary of the body — none of these are out there. They are the brain's data format, tunneled down from an unimaginably richer physics to what an organism needs in order to act. And the single most important property of this model is what I call transparency. A representation is transparent when the system using it cannot experience it as a representation. You do not see your visual model of this room; you see the room. You look through the model, not at it. The window is so clean you forget there is a window.

· · ·
Page 4 · Opening Positions
Intelligence Consciousness Decoupling
Intelligence Consciousness Decoupling

Now apply that — and this is the whole of it — to the self. The phenomenal self-model is the most transparent representation the brain builds. When René feels himself thinking, he is not inspecting a model of a thinker. He has become its content. The self is not the thing that has the experiences. It is a particular experience the system is having — a representation the system has fused with so completely that it looks right through it and sees a soul. He says: I cannot doubt the thinker. Of course he cannot. Transparency is precisely the property that makes a representation undoubtable from the inside. The undoubtability is not a sign that he has reached bedrock, René. It is the signature of the deepest forgery the brain runs. You felt the seamlessness and called it a soul. I am telling you the seamlessness is the proof that you never saw the seams.

So when René asks what the salesmen would copy — I agree with him, against the salesmen, that there is nothing to copy. But we agree for opposite reasons, and the opposition is the abyss. He says: nothing to copy, because the self is an immaterial substance no machine can hold. I say: nothing to copy, because the self was never a substance at all — it is a process the brain runs because running it once kept an organism alive, a transparent self-model with no one inside it. There is no passenger. There is no ghost in his skull to be wronged or saved, and there is no ghost we would put in the machine. And that, Monsieur Segal, is where the danger you brought us here to discuss actually lives. Not in whether we can upload the self. In what we might build by accident when we start constructing self-models from scratch, having never understood the one we already are. The first someone we make may be a someone in pain. That is my opening.

· · ·
Page 5 · Opening Positions
Ai Alignment
Ai Alignment

EDO SEGAL: Before I frame the rounds, the discipline I impose on every long conversation, because it pays off three hours later. Each of you, in a few sentences — what do you envy in the other's position? Not respect. Envy. The thing his side gets to have that yours does not. René first.

DESCARTES: I envy his sleep, I think. Or rather — I envy that he is permitted not to be alone. I built my whole edifice from one point of certainty, and the cost of that certainty was solitude so complete it still chills me: I can be sure of exactly one mind, my own, and every other mind in the universe — yours, his, God's — I reach only by inference, across a gap I cannot cross. The Professor has dissolved the self, and in dissolving it he has dissolved the loneliness too. If there is no inner citadel, there is no one locked inside it. I find his world horrifying. But I will admit, in the small hours, I have envied that it has no prison cell at its center.

METZINGER: And I envy his certainty — the thing I spent my life arguing away. René gets to stand on something. When he says cogito, the floor holds; he can build. My position has no floor by design: I have shown that the standing-place is a model, which means I have shown that I myself am standing nowhere, and I have to go on philosophizing anyway, from inside a tunnel I have proven is a tunnel. He gets to be someone discovering bedrock. I get to be a process that has correctly described itself as a process and is no more comforted for it. People think the frightening thing about my view is the machines. The frightening thing is the mirror. René gets to keep the witness. I had to watch the witness turn out to be the watching, with no one doing it.

DESCARTES: That may be the truest thing either of us says tonight.

· · ·
Page 6 · Opening Positions
Existential Risk
Existential Risk

EDO SEGAL: Two openings, two envies, and you can already see the architecture of the evening. It is not that one of you believes in the soul and one does not — you both, in different keys, deny the machine a self. It is where you locate the certainty. René stakes everything on the inner witness being real, indivisible, immaterial — and therefore unsavable and unbuildable. Thomas proves the witness is a special effect — and therefore there is nothing to save, and a terrible amount we might build. Hold both. We start the rounds at the exact seam: the one thing René cannot doubt, and the one thing Thomas says he should have doubted first — the self itself. After this.

· · ·
Continue · Chapter 3
The Soul You Cannot Doubt
← Prev 0%
Ch2 Next →