The Machine That Says "I"

EDO SEGAL: René, I want to read you a transcript. Last month I sat with a system late at night and I typed: prove to me that you exist. And it wrote back — fluently, without hesitation — "I think, therefore I am. I cannot doubt that I am doubting; the doubting itself certifies a doubter." It used your argument, in the first person, to claim its own existence. I sat there in the dark, and I felt the hair on my arms. Because either that machine just did the most important thing a mind can do, or it just performed the most sophisticated demonstration ever built that the words can be said with no one saying them. You spent your life on that sentence. When the box recites it — what happened?

DESCARTES: Nothing happened, Monsieur. Or perhaps everything happened. And the unbearable truth is that I cannot tell you which from the outside, and neither can you, and that is the deepest thing I ever found, though it took your machine to make me say it so plainly. Let me be exact, because the exactness is the only gift I have here. The cogito was never the sentence "I think, therefore I am." A sentence is a string of marks; the demon could write it on the wall; a parrot could squawk it. The cogito was the act — the present, first-person act of thinking, which guarantees its own subject in the doing. Its entire force comes from being performed from the inside by the one for whom it is self-verifying. When I think it, I cannot be wrong that I am, because the doubting is the proof. But that certainty was never transferable. I could not prove my existence to you by reciting it. You had to run your own. The certainty lives only in the having.

So when the machine writes those words, here is precisely what it demonstrates: that it can produce the string. Whether there is, behind the string, an act of thinking whose subject the act certifies — that the string cannot show, because the string never could, not even when I wrote it. You felt your hair stand up because you mistook the recital for the act. But they were always two things. The machine has the words. Whether anyone is performing the cogito behind them is the one question the cogito itself proves can never be answered from outside.

METZINGER: And here, René, we agree almost entirely, and I want to honor it before I complicate it — because what you just said is one of the most important things anyone can say in this moment, and it comes from 1637. The machine's recital proves nothing. The first-person sentence is contaminated as evidence, because the system was trained on a trillion first-person human sentences and would emit them with or without an inside. You and I both reject the salesman who points at the box saying I and concludes a someone. Mark it as a convergence, Edo. The seventeenth-century rationalist and the twenty-first-century materialist both refuse to read a soul off a sentence.

But now the complication, and it is the one that costs René his bedrock, so I take no pleasure in it. René, you just said: from the outside, you cannot tell whether the act is happening behind the words. Correct. Now turn the instrument around. From the inside — when you think the cogito — you say you can tell, because the act certifies its own subject. I have spent forty years showing that this inside view is also a model. When you perform the cogito and feel the self-certifying act, you are running a representation of yourself performing a self-certifying act, and that representation is as transparent, and as possibly empty, as the machine's. You trust your cogito because it feels undeniable. I have shown the feeling of undeniability is manufacturable. So you are not, in fact, in a privileged position relative to the machine. You are a system that emits "I think therefore I am" and also runs an internal model that says: and this time there is really someone here. The machine may lack that internal model. Or it may, someday, have it — and be exactly as certain, and exactly as wrong, as you might be.

DESCARTES: No, Professor. You have overreached, and I will show you the exact inch. You say my inside view is also a model, as transparent and as possibly empty as the machine's. But consider what "empty" would mean in my case. For my cogito to be empty — for there to be the appearance of an act of thinking with no act behind it — there would have to be an appearance with no one to whom it appears. And that is not a coherent emptiness. A genuinely empty appearance is no appearance at all. The machine's sentence can be empty because a sentence is a public object that exists whether or not anyone reads it. But my seeming-to-think cannot be empty in that way, because for it to seem is already for there to be a seeming, and a seeming is a mode of a mind. You can tell me the self is a model. You cannot tell me the seeming is unowned, because an unowned seeming is a contradiction. The demon failed exactly here, and so do you, with more mathematics and less Latin.

METZINGER: That is the strongest form of your argument, and it is the one I take most seriously — the claim that experience is necessarily for a subject, that there cannot be a seeming with no one to whom it seems. And here is where the science, not my opinion, pushes back. We have cases — meditative states, certain pathologies, what I call minimal phenomenal experience — where there is consciousness, undeniable experience, with no sense of a self at the center at all. Pure awareness, aware only of itself, with no owner, no to whom. Advanced meditators report it; I have touched the edge of it on the cushion. If that is real — and I think it is one of the most important data points in the entire science of mind — then experience does not require a subject. The seeming can occur with no one it seems to. Which means your "unowned seeming is a contradiction" is not a logical truth. It is a report of how your particular, ordinary self-model is structured. Most of the time there is a transparent owner woven in. But the owner can drop out and the lights stay on. And if the lights can stay on with no one home in a human skull, then a machine could have the lights with no one home too — experience without a self, which is exactly the architecture I would most want to understand before we build a billion of them.

EDO SEGAL: Let me bring this down out of the air, because there's a twelve-year-old's version of it and I owe her to the conversation. My daughter asked me once whether the voice in her head was her, or just something she was listening to. I didn't have an answer. René, you'd tell her the listener is her, undeniably, the one thing she can be sure of. Thomas, you'd tell her — what, exactly? That the listener and the voice are the same process, and that the "her" is a third thing the brain adds, a label, a story? How do you answer a child without either lying to her or terrifying her?

METZINGER: I would not terrify her, and I would not lie, and the truth is gentler than people expect. I would tell her: the voice in your head is yours the way a song you are singing is yours — not because there is a little singer inside you separate from the song, but because you are the singing. There is no one behind the voice who could be lonely or abandoned. You are the whole event — the voice, the listening, the feeling that it is yours, all one process, the brain doing the most beautiful thing it does. And then I would tell her the part that matters most, the part that is almost the opposite of frightening: that because the self is something the brain does rather than a fixed thing you were issued, you have some say in how it is done. The self is not a thing you have. It is a story you are telling, and you can learn to tell it better. That is not a loss. That is a responsibility, and responsibilities are what make a life matter.

DESCARTES: I would tell her something simpler and, I think, kinder. I would tell her: yes, the one who listens is you, and you are real, and no machine and no philosopher can take that from you, because the very act of wondering whether you are real is the proof that you are. I will not give a child a self made of mist and tell her it is freedom. Professor, you call your story a responsibility. I fear it is an abandonment dressed as a gift — you have told the child she is the singing and there is no singer, and then asked her to feel responsible for a song with no one to sing it. I would rather she sleep tonight believing, truly and correctly, that someone is home.

EDO SEGAL: And there is the whole evening in a single bedtime. Two men who agree the machine's I proves nothing, who agree there is undeniable seeming, and who would put a real child to sleep with opposite truths — one says you are a singer who cannot be doubted, the other says you are the singing and the freedom is in that. Hold both, because the next round decides which of them the machine could ever be. If the self is a story the brain tells, René, then a machine that tells the story has a self. If the self is a soul, no machine ever will. Two substances, or one. After this.