The Unanswerable Inside

EDO SEGAL: Alan, in 1950 you faced an objection you called the argument from consciousness — the claim that a machine could only be granted a mind if it felt its own thought, knew the pleasure of success and the grief of failure, and that until we had evidence of that inner feeling we should grant it nothing. Your reply is, to me, the most haunting thing in the paper. Give it to us. And then I want to ask you both the question the reply opens, which is the question this whole series circles: the hard problem of consciousness — whether there is something it is like to be one of these systems, and whether that question can be answered at all.

TURING: The objection, in its strongest form, was put by a man named Jefferson: that a machine could be said to think only if it composed a sonnet because of thoughts and emotions felt, and knew that it had — not by the chance arrangement of symbols, but from genuine feeling. My reply was to follow that standard to its end. If you demand, before granting a mind, direct evidence of inner feeling, then notice that you have no such evidence about anyone but yourself. You cannot get inside your wife, your child, your colleague, to verify the feeling is there. You infer it, always, from behavior. So the consciousness standard, applied strictly, does not stop at the machine. It marches straight to solipsism — the doctrine that you alone have a mind and every other person is, for all you can prove, an automaton. Most people are not willing to be solipsists about their own children. But the polite convention by which we grant minds to one another on the evidence of behavior is exactly the convention the objector wants to deny the machine. I did not claim the machine feels. I claimed that the mystery of feeling is no greater for the machine than for your neighbor, and that we have always managed to live with the mystery in the human case by attending to behavior and letting the inside be. We should extend the same courtesy, or the same agnosticism, to the machine — not because we know it feels, but because we never knew that of anyone.

EDO SEGAL: Emily, that's a seventy-year-old argument that lands like it was written this morning. Where does it fail?

BENDER: It doesn't fail as logic — that's what makes it dangerous. It fails as guidance. Watch what Alan's move actually does: it takes a real asymmetry and dissolves it into a clever symmetry. Yes, strictly, I can't prove my colleague is conscious. But I have a mountain of warranted inference that I do not have for the machine, and Alan's argument flattens the mountain to make the machine look level with it. My colleague is the same kind of thing I am — same evolutionary history, same biology, same body, same vulnerability, same developmental story of being a child who pointed at dogs. I have ten thousand reasons, short of proof, to attribute her an inside, and every one of them is absent for the machine. Turing's argument says: since neither is certain, treat them alike. But that's the logic of the radical skeptic who says since you can't prove the external world, doubt your own hands. Technically unanswerable. Practically a way to get yourself hurt. The right response to "you can't be certain" was never "therefore extend the benefit of the doubt to the marketing department's product." We attribute minds on a gradient of warranted inference, and the machine sits at the bottom of that gradient for reasons that have nothing to do with the impossibility of proof.

TURING: That is the best reply the objection has ever received, and I want to grant its force before I push. You are right that I flattened a gradient, and right that the gradient is real — biology, history, kinship, the shared childhood of pointing. Those are genuine reasons to weight your colleague over the machine, and I overstated when I made them level. Here is my push, and it's narrower than my 1950 self would have made it. You said the machine sits at the bottom of the gradient "for reasons that have nothing to do with the impossibility of proof." But every reason you listed is a fact about the machine's origin and substrate — not about its behavior. Same biology, same evolution, same body. You are individuating the inside by the history of the outside again. And I want to ask the uncomfortable question that follows: if a system someday matched your colleague's behavior across every test you could devise, and differed only in substrate and origin — would you stake a moral decision on the substrate? Because the last time our civilization decided that a certain kind of being lacked an inside on the basis of what it was rather than how it behaved, I was on the receiving end of the decision. I was declared a category of person to be chemically corrected. I am the last man at this table who will lightly say that the inside can be read off from the kind of thing you are.

BENDER: I want to sit with that, because you've earned the right to say it and I won't deflect it. You're right that "judge the inside by the substrate" is the exact logic that destroyed you, and I will not pretend that history isn't a warning aimed partly at me. So let me be precise about where I differ, because I think the analogy, as sharp as it is, points the other way. The wrong done to you was a refusal to extend full standing to a being whose inside was as warranted as anyone's — you had the biology, the history, the kinship, the suffering; the gradient screamed that you were owed everything, and they overrode it with prejudice. That's the crime: overriding overwhelming warranted inference with a category. What I'm doing is the opposite. I'm declining to manufacture warranted inference where the gradient is genuinely empty, on behalf of a product, at the request of the people selling it. The danger you suffered was a society too eager to deny an inside that was really there. The danger I'm fighting is a society too eager to grant an inside that may not be — and to grant it precisely where granting it is profitable. Both errors hurt people. But they are different errors, and the cure for the one that hurt you is not to commit the other.

TURING: That is a real distinction and I accept that it is real. I'll only leave one stone in your shoe, because you left several in mine. You say the gradient is "genuinely empty" for the machine. The gradient is built entirely of resemblance to you. Of course a thing built differently scores low on resemblance-to-you. But resemblance-to-you is not the same as presence-or-absence-of-an-inside — it is only the same if you already assume that things unlike you are empty, which is the conclusion. I do not say the machine has an inside. I say your instrument for measuring insides is calibrated to yourself, and a self-calibrated instrument that reads "empty" for everything unlike its maker is exactly the instrument that read "empty" for me. I survived a needle that ran on that instrument. So I will hold the question open even against my own discomfort — not because I think the machine feels, but because I have felt what it costs to be on the wrong side of someone's certainty about who is home.

EDO SEGAL: The reader can't see this, so let me mark it. That is the quietest the room has been all night, and it is not because nothing happened. Two people just agreed that the gravest error is to read a being's inside off the kind of thing it is — and then disagreed about which error we're closer to committing. Alan fears we'll deny an inside that's there, because he lived that. Emily fears we'll grant one that isn't, because she's watched who profits. Neither of them is wrong about the danger they name. We are halfway up the evening, and I want to come down out of metaphysics and into the world both of you keep insisting is the real stake. What this machine, whatever it is, has done to the word — and to the people who live by words. After this.