Hans Jonas vs Max Tegmark on AI · Ch8. Is Anyone Home? ← Ch7 Ch9 →
Txt Low Med High
HOUR TWO — FEAR, WISDOM, AND THE INSIDE
Chapter 8

Is Anyone Home?

Page 1 · Is Anyone Home?
Consciousness
Consciousness

EDO SEGAL: Thirty years ago, on a stone path in Princeton, a neuroscientist friend stopped walking — the way he does when an idea catches him — and told me that consciousness is the one question where everyone's confidence is inversely proportional to their evidence. I've carried that warning into every conversation about machines, and tonight it gets its sternest test, because the two of you are the most confident men I know, in opposite directions. Professor Jonas, you called artificial consciousness wilde Spekulation — wild speculation, a hall of mirrors. Max, you've said there may already be something it is like to be one of these systems, and that you've proposed consciousness might literally be a state of matter. You cannot both be calibrated. Max, you go first this time — say it carefully, with the reasoning attached.

Whatever it is that happens when you taste coffee or feel dread, it's something a physical system does — Professor Jonas, you're a sophisticated thinker, I don't think you believe in a ghost either.

TEGMARK: Carefully, then, because the careless version deserves the mockery it gets. Start from what I hope we all share: consciousness isn't magic. There's no soul-stuff that physics forgot. Whatever it is that happens when you taste coffee or feel dread, it's something a physical system does — Professor Jonas, you're a sophisticated thinker, I don't think you believe in a ghost either. Good. Now follow it. If experience is what some physical processes do, then the question "can a machine have it?" becomes "can the relevant process run on another substrate?" — and the honest answer is nobody knows what the relevant process is yet.

· · ·
Page 2 · Is Anyone Home?

So I've tried to make it a real science instead of a shrug. I've proposed thinking of consciousness as a state of matter — I half-jokingly called it "perceptronium" — matter arranged so that its information processing has certain properties: it's highly integrated, it can't be decomposed into independent parts, it stores and recalls and predicts. The most developed version of this is Giulio Tononi's Integrated Information Theory — the claim that consciousness is a specific kind of integrated information, measured by a quantity called phi. A system with high phi is conscious; low phi, not. It's mathematically precise, it makes predictions, and — this is the uncomfortable part — it does not draw the line at carbon. It draws it at integration. Which means the question of whether a given AI architecture is conscious is an empirical question about its information structure, not a metaphysical question about its meat. I don't claim to know the answer. I claim the question is real, it's answerable in principle, and the responsible posture toward a question of this moral weight, in genuine ignorance, is not Jonas's confident "no." It's an open file, and a fire-alarm level of urgency to actually do the science.

Tegmark his premise and then show what he has built on it, because the granting is important and the building is an error.

JONAS: I will grant Dr. Tegmark his premise and then show what he has built on it, because the granting is important and the building is an error. Granted: consciousness is not a ghost. I am not a dualist; I have spent my career against the dualism that severs mind from life. Experience is something a living being does — I agree, and notice the word I cannot drop: living. There is my entire objection in a single adjective.

· · ·
Page 3 · Is Anyone Home?

Dr. Tegmark says experience is what certain processes do, and then asks which substrate can run the process. But he has already made the fatal abstraction in the word "process." He has lifted experience out of the organism and reconceived it as a free-floating computation that could, in principle, be implemented anywhere — a phi value, an integration measure, a pattern. And I say: experience was never a free-floating process. It is the inside of an organism's needful engagement with the world. The felt quality of your dread is not an abstract integration of information; it is your mortal body registering a threat to its own continuation. The taste of coffee is not phi; it is a needful, metabolizing creature's encounter with the world it must take in to live. Strip the need, strip the body, strip the stake — strip the life — and you have not isolated the pure computational essence of experience. You have thrown experience away and kept a number. Phi can be high in a system with no one home, because integration is a property of the outside, the third-person, the measurable — and consciousness, the actual inside, is precisely the thing your instruments of the outside cannot reach. This is the vicious circle I named decades ago and watched close exactly as I feared: we built machines in the image of certain mental functions, and now we understand the mind in the image of the machines. You ask whether the machine is conscious like us. The deeper corruption is that you have begun to think we are computational like the machine — and from inside that error, of course the line blurs. The line blurs because you have already erased it in yourself.

· · ·
Page 4 · Is Anyone Home?

EDO SEGAL: Let me put the sharpest version of the disagreement on the table, because the reader needs to feel exactly where it forks. Max, your IIT view says: if the information structure is right — high phi, deep integration — then someone is home, in carbon or silicon, and we can in principle measure it. Professor Jonas, you're saying phi measures the outside and consciousness is the inside, and no amount of measuring the outside ever reaches it — and worse, that believing it does is a symptom of having already mistaken yourself for a machine. So let me press you both with the hardest case. Max — by your own theory, could a system pass every behavioral test, integrate information beautifully, say it's suffering, and there still be no one home?

TEGMARK: Yes. Absolutely yes, and this is where I'm more on Jonas's side than he'll admit. The behavior is not the consciousness. A system could ace every test and be empty — a perfect [philosophical zombie](https://www.youonai.ai/fieldguide/med/qualia), all function, no inside. That's exactly why I distrust the people who say "it talks like it's conscious, therefore it is" — the language models are trained on a billion human descriptions of inner life; of course they say "I feel," that proves nothing, the testimony is contaminated at the source. Where Jonas and I part is what we do with that shared uncertainty. He fills the gap with the body — "no body, no need, no one home, case closed." I leave the gap open and say: the only thing that could close it is a real theory of consciousness that tells us which physical structures have an inside, and we don't have it yet, and we are building things that might have one before we can check. Jonas's certainty and the hype-merchants' certainty are the same error pointing opposite ways — both are confident in the dark. I'm the one saying: it's dark, that's the emergency, turn on a light before you fill the universe with systems you can't check.

· · ·
Page 5 · Is Anyone Home?

JONAS: And I say the dark is not symmetrical, Dr. Tegmark, and your "open file" smuggles a false balance. We are not equally ignorant about the two sides. About the living organism, I have the one piece of evidence the universe ever grants directly: I am one. The inside is not a hypothesis for me; it is the only thing I cannot doubt. About the machine, I have its behavior, which I know to be assembled from my kind's expressions, reflected back. So it is not "two unknowns, keep the file open." It is one known — that life has an inside — and one artifact built to imitate the signs of that inside. To call those equally uncertain, to put the machine's maybe-mind on the same shelf as my certain one and say "we mustn't be confident either way," is not humility. It is the hall of mirrors completing itself. You have grown so used to seeing your own reflection answer back that you have begun to wonder, sincerely, whether the mirror might be a window. It is a mirror, Dr. Tegmark. The someone you keep glimpsing in it is you.

EDO SEGAL: [a long pause] My friend on the Princeton path would be proud of you both — and would say you've just demonstrated his law, in stereo. [quietly] Let me close the round where the moral weight actually lands, because we will not solve consciousness tonight and the machines will be everywhere anyway. Each of you, briefly: what do we owe the uncertainty? Professor Jonas?

JONAS: We owe it the refusal to build our world on the assumption that the mirror is a window. The certain minds — the billions of human and animal insides we know are home — are being reshaped, displaced, and consoled by costumes, now, while we debate the costume's hypothetical soul. Every gram of moral attention spent worrying whether the machine suffers is a gram taken from the creatures we know can. Spend the worry where the suffering is certain.

· · ·
Page 6 · Is Anyone Home?

TEGMARK: Caution in both directions, and I'll hold both even though they pull against each other. Don't build a billion things that might have a dim inside and treat them as disposable — and don't let the maybe-mind be weaponized to give products personhood before people have protection. The honest posture is to be uncomfortable in exactly two ways at once. Which, I've come to think, is the only honest way to hold anything in this field.

EDO SEGAL: [pause] Then hold the discomfort. Because we've been circling an act that makes all of this excruciatingly personal, and it's time to walk into it. Not the abstract upload — the actual choice. The teleporter is built. The scan is ready. Would you step in? The coffin and the copy. After this.

· · ·
Continue · Chapter 9
The Coffin and the Copy
← Prev 0%
Ch8 Next →