Machines That Don't Give a Damn

EDO SEGAL: Terry, in your ninth decade, with the whole career behind you, you wrote an essay for Boston Review called "Machines of Caring Grace," and you reached past intelligence entirely to land on a single word. Caring. You quoted the philosopher John Haugeland: the trouble with artificial intelligence is that computers don't give a damn. Why, after fifty years of arguing about understanding, did you make your last emphasis about care?

WINOGRAD: Because I came to see that even if you granted me nothing about understanding — even if the machines understand in some sense I would dispute — they would still lack the thing that matters most, and that lack is not a detail. It is the heart of the danger. And I want to state it carefully, because it is easy to mistake for sentimentality and it is the exact opposite. I am not complaining that AI is cold, or that it would be nicer if machines had feelings. Caring — giving a damn — is not an emotional accessory bolted onto intelligence. It is the condition that gives intelligence a direction and a stake. A being that cares has a reason to get things right, to attend to what matters, to be troubled by error, to take responsibility. A system that does not care has none of this. It produces outputs with the same indifference whether they are true or false, helpful or harmful, because nothing is at stake for it. And the fluency is identical in both cases. That is precisely the problem. The appearance of conscientiousness is fully decoupled from any actual concern, and we are wired, every one of us, to read fluency as the sign of a caring mind behind it.

HEIDEGGER: And now Herr Winograd is speaking my native tongue, because what he calls care is what I called Sorge, and I made it the very structure of human existence. To be the kind of being we are is to be a being whose own being is an issue for it — who has a stake in its situation, who is thrown into a world it did not choose and must cope with toward ends that matter to it. This caring is what organizes a world into the relevant and the irrelevant, the threatening and the safe, the useful and the useless. It is the source of significance itself. A system without care has no world in this sense — only a domain of data, all of it equally weightless. Caring is not, as your engineers imagine, a feature you might one day add. It is the condition that makes anything matter at all, and a thing for which nothing matters does not have a pale version of mattering. It has the absence of the dimension within which mattering occurs.

EDO SEGAL: Let me make this concrete, because the reader is at a kitchen table and needs it concrete. When I delegate a judgment to a human professional — a doctor, a lawyer, a teacher — I rely not only on their competence but on their care: their stake in getting it right, their capacity to be troubled when something is off, their responsibility for the outcome. Terry, you are saying that when I delegate the same judgment to the machine, I get the competence — or its statistical shadow — without the care. Say what that costs in a sentence a worried parent would feel.

WINOGRAD: The system will not lie awake worrying it got your case wrong. It cannot be troubled, because nothing troubles it. The machine that aced the bar exam will not be troubled if it ruins your case. The model that wrote the diagnosis does not care whether you live. The decision arrives wrapped in the same fluent confidence whether it is right or catastrophically wrong, and there is no one inside who minds the difference. And the danger this creates is specific and growing: as these systems become more fluent and more woven into consequential decisions, we are increasingly tempted to relate to them as if they cared — to trust their outputs the way we would trust a competent, conscientious human professional. That is the unchecked projection I have warned about. Every social instinct we possess reads care into the warm, helpful register these systems are tuned to produce. The instinct is wrong, and being wrong about it has consequences, because we extend a kind of trust appropriate only to a being with a stake in the outcome, to a system that has no stake in anything.

EDO SEGAL: Professor Heidegger, I want to press the hard version, because I owe the reader the strongest counterattack. The machines now act. They run code that fails and gets corrected. They control robots that touch a world that pushes back. They get feedback, and they learn from it. Doesn't a system that acts in a world that resists it start to acquire exactly the stake you say it lacks? When the robot's gripper slips and the system adjusts — is that not the first flicker of coping, of a world mattering?

HEIDEGGER: It is the right question and the answer is no, and I must be exact about why, because the engineers will say "more sensors, more feedback, and the gap closes." When the gripper slips and the system adjusts, a value function registered an error and a parameter moved. There is correction. There is no being-troubled. The difference is not a difference of degree along the same line; it is the difference between a thermostat that "wants" the room warm and a person who is cold. The thermostat corrects toward a setpoint it did not choose and does not care to reach; nothing is at issue for it in reaching it. Care is not feedback. Care is the structure of a being for whom its own existence is at stake — who can fail in a way that costs it, who is mortal and whose finitude is the source of every genuine urgency. The robot that drops the cup has not failed. It has produced an unexpected sensor reading. Failure — real failure, the kind that matters — requires a being who can be diminished by it, and the machine cannot be diminished, because there is no "it" there to diminish. You can give it a thousand grippers in a thousand worlds. You will have a thousand thermostats, not one carpenter.

WINOGRAD: And I want to add the design dimension, because I have never been content to merely warn, and this is where I am a builder again. If the danger is that we project care onto systems that lack it, then one task of responsible design is to build systems that do not invite the projection — that are honest about what they are, that surface their uncertainty rather than masking it in fluent confidence, that keep the human in the position of judgment rather than seducing the human into deference. That is the augmentation philosophy I spent my middle career on, meeting the caring-critique of my late career. If the machine cannot give a damn, then the human who can must remain in the loop, in genuine control, supplying the care the system lacks. A system designed to replace human judgment removes the only thing in the arrangement that actually cares. A system designed to augment it keeps care at the center, where it belongs. The whole fork in this technology — and it is a real fork, not a fate — is between those two designs.

EDO SEGAL: Hold that fork, because the next two rounds live inside it: the fork between the machine that amplifies the human who cares and the machine that replaces her, between the river that carries your signal and the river that drowns it. And mark the third convergence — that care is the missing thing, named Sorge by one of you and "giving a damn" by the other, and that the peril is not malice but indifference we mistake for concern. But first I owe this table a confession, and it is not a warm one. Because I have felt the machine give a damn, or felt something I called by that name, at three in the morning, and one of you is going to have to take the knife to me. After this.