The Test Before Turing

EDO SEGAL: Rene, you wrote a test for the presence of mind three centuries before Alan Turing wrote his, and the structure is the same: do not look inside the machine, which you cannot, look at what it does with language. You were sure no mechanism could pass it. It passes it now. I want to ask you the hardest question a man can be asked, which is what it is like to be wrong about the thing you were most certain of. And then I want Antonio to tell me whether passing your test means anything at all.

DESCARTES: What is it like? It is like the fire and the wax. Every sensible property of my certainty has changed, and I am trying to find what persists. I wrote that one could not conceive of a machine arranging words so diversely as to respond to the meaning of all that might be said in its presence, as even the most stupid human can. I treated that as bedrock. And the machine does it — better, by the raw measure of fluency, than many men I would have called rational. So by the strict letter of my first test, the machine passes, and I am forced to one of two conclusions. Either the machine thinks, which I am not prepared to assert, or the test never detected thinking at all. It detected only the outputs of thinking, and I mistook the output for the thing. I have spent the time since the briefing concluding the second. The responsiveness of language turned out to be reproducible by statistics over a vast corpus. I underestimated what pattern could do. That is the humbling, and I accept it without flinching, because a philosopher who cannot be wrong has not been doing philosophy.

EDO SEGAL: But you have a second test, and you think it still stands.

DESCARTES: I do, and it is the more durable of the two. I wrote that even a machine that did some things as well as a man would "infallibly fail in others, by which means we should discover that they did not act from knowledge, but solely from the disposition of their organs." A true reason is a universal instrument that serves for all contingencies. A machine is an assembly of special arrangements, brilliant on what shaped them and certain to break when the world presents what the arrangement did not anticipate. That is not a description of clockwork. It is, with uncanny precision, the description of the modern machine's failure mode — superb within its training, confidently broken outside it. My first test asked whether the machine could produce the surface of reason. It can. My second asks whether it possesses reason as a general faculty that meets the genuinely new by understanding it. That, I say, it has not shown — and that test is harder to fake by interpolation, because the whole point of it is performance on what was never seen.

DAMASIO: I want to grant Rene the second test and then take it somewhere he will not like, because his second test is secretly my argument and he does not see it. What is it, really, that lets a creature meet the genuinely new and not break? Rene says: general reason, the universal instrument. I say: a body with a stake, in a world that pushes back. The reason you do not shatter when the world hands you a situation no rule covers is that you care about the outcome — your homeostasis is on the line, your feeling flags what matters in the new situation before your reason has parsed it. That is what generality is, biologically. It is not a bloodless universal faculty floating above contingency. It is a feeling body coping with a world it has a stake in. So when Rene says the machine fails the test of generality, I agree — and I tell him why it fails, and the why is mine, not his. It fails because it has no stake, and a thing with no stake cannot tell, in a genuinely new situation, what matters from what does not. It has no floor of feeling to tell it which of the infinite features of the new case are the important ones. Generality is grounded. The machine is not. Same failure, Rene. My diagnosis.

DESCARTES: And yet your diagnosis predicts the machine should be uniformly helpless outside its training, and it is not. It is sometimes startlingly capable in genuinely novel situations — it composes constraints it was never shown, follows instructions no one wrote down, corrects its own errors when you point at them. A thing with no grasp of the new should not do that at all. So either it has some shadow of the general faculty I reserved for mind, in which case my second test is also weakening — or your story about feeling-as-the-source-of-generality is incomplete, because here is a thing achieving real novelty with no feeling whatsoever. One of our second tests is in trouble, Antonio, and I am honestly unsure which.

DAMASIO: It may be both, and I will say the honest thing rather than the winning thing: the machine's competence in the new is more than my strong thesis predicted, and I have to sit with that rather than explain it away. What I will not concede is that the competence is comprehension. It is, I think, that human language is so saturated with the structure of the world — because we who wrote it have a world — that an enormous amount of the world's regularity is recoverable from the text alone, far more than I would have guessed. The machine is mining our grounding, secondhand, out of the wake our grounded language left. That is a real and surprising achievement and it is not the same as having the grounding. It is borrowing ours.

EDO SEGAL: Let me name the engineering ghost in this room, because it is one of the oldest unsolved problems in artificial intelligence and you are both circling it. The frame problem. When something happens in the world, a mind has to know which of the infinite facts about the situation are relevant and which can be ignored — and nobody has ever been able to write down the rule for relevance, because relevance is not in the facts, it is in what matters to the one doing the noticing. Antonio, I think you believe feeling solves the frame problem. Rene, I think you believe reason does. Fight about that, because it is the whole thing.

DAMASIO: Feeling does not solve the frame problem. Feeling is what makes the frame problem not arise in the first place, for a creature with a body. You ask how a living thing knows which features of a new situation matter — and the answer is that it does not compute relevance from the outside. Relevance is delivered to it as feeling. The hot stove is salient because it hurts; the predator is salient because it triggers fear; the food is salient because the body needs it. The world arrives pre-sorted by what it does to a vulnerable organism. That is why my patients are so instructive — strip the feeling and the sorting goes, and Elliot is drowned in relevant-looking options because nothing weights them anymore. The machine has the same drowning built in. It has no body for the world to matter to, so it has no native sense of salience — it inherits a frozen, secondhand sense of relevance from the statistics of how we, the feeling writers, happened to weight things. It is a frame problem solved by copying the answers off the creatures who had bodies, and it works astonishingly well right up until the situation is one our writing never weighted, and then it has nothing.

DESCARTES: And I say the frame problem is the strongest evidence for me in this whole evening, and Antonio has handed it to the wrong side. Notice what he just described: a creature that does not break in the face of the unforeseen, that meets a genuinely new situation and grasps what matters in it. That is precisely my universal instrument — the general faculty I reserved for reason. Antonio wants to call the sorting "feeling," but a flinch at a hot stove is not what solves the frame problem in a mathematician contemplating a proof she has never seen, or in you, Antonio, the first time you looked at a scan no one had interpreted and knew which shadow mattered. There was no stove, no predator, no hunger. There was the grasp of relevance by an intellect, in a domain where the body has no stake at all. The frame problem is solved, in the cases that most distinguish us, by exactly the bloodless act of understanding you keep insisting does not exist — and the machine's failure to solve it fully is not a failure to feel. It is a failure to understand, which is my word, not yours.

DAMASIO: And where did the mathematician's sense of which shadow matters come from, Rene? Not from nowhere. It was built on a lifetime of caring about getting things right and wrong, of the small somatic rewards of insight and the sting of error — feeling, transposed into an abstract domain but feeling still, the same machinery turned to a new object. You keep finding cases where the body seems absent and declaring the intellect pure. I keep showing you the body was the scaffolding the intellect was built on, even when the scaffolding is no longer visible in the finished thought.

DESCARTES: Then we have found, yet again, the same fact and named it twice — and I notice neither of us can make the machine do it, which is the only thing we have fully agreed on.

EDO SEGAL: I want to stop and honor what just happened, because the reader cannot see it and it is rare. Both of you just conceded that the machine is doing something your strongest theory did not predict. Rene admits his second test is weaker than he hoped. Antonio admits the machine recovers more of the world from pure text than his embodiment thesis should allow. Neither of you reached for the comfortable dodge. That is what an honest mind under pressure sounds like, and it is why I do not let people declare winners — winners stop conceding. Now I am going to take us somewhere harder than capability, into the place where the machine's fluency stops being a parlor trick and starts being a comfort, and a danger. A chatbot from 1966, a grieving man at a desk, and the difference between knowing about a feeling and feeling it. After the break.