Replacing Guilt

In the [YOU] on AI Field Guide

The [YOU] on AI cycle is animated by the conviction that the right response to the arrival of capable machines is not panic or denial but the discipline of clear sight, sustained over time. Replacing Guilt is Soares's account of how that discipline is maintained psychologically. A person who has absorbed the possibility that the default trajectory leads to civilizational catastrophe faces an obvious psychological hazard: such a belief could easily become paralyzing, a source of despair or frantic guilt-driven overwork. The Replacing Guilt philosophy is the emotional counterpart to Soares's technical pessimism, the framework that lets him hold his catastrophic conclusion without being unmade by it.

The series is also the place where Soares's alignment thinking turns inward most explicitly. Just as he worries that an AI's behavior will diverge from the values its makers intended—driven by training pressures rather than genuine alignment with the goal—he worries that a person's effortful behavior will diverge from the values the person actually holds, driven instead by a guilt that serves no one. The remedy in both cases is alignment in the literal sense: bringing behavior into accord with genuine value. The difficulty in both cases is that optimization pressures (in the AI case, gradient descent; in the human case, cultural conditioning and the fear of social judgment) do not reliably produce this alignment, and the conscious work of recovering it is non-trivial.

Origin

Soares began writing the essays in 2014, shortly after leaving Google to join MIRI, as a way of working through the psychological challenge of committing to what he understood as the most important problem in the world in the face of both high uncertainty about its tractability and deep uncertainty about his own effectiveness. The title captures the series' central move: not the elimination of moral seriousness but the replacement of a particular motivational structure (guilt, obligation, the sense of falling short of a demand) with a different one (intrinsic drive, genuine care for the goal, sustained effort from commitment rather than compulsion).

The series draws on Soares' background in decision theory, applying to the personal domain the same analytical precision he brought to questions about rational agency and instrumental rationality. It is also the place where his thought about functional decision theory connects to ordinary psychology: the question of how an agent's choices relate to its values is, in the human case, not merely a formal puzzle but a practical challenge that most people navigate badly, and the series is a guide to navigating it better.

Key Ideas

Guilt as a bad motivational engine. Soares argues that guilt-based motivation is intrinsically self-undermining: it treats the agent as a source of obligation to be managed rather than as a person with values to be expressed. It produces overwork in the short run and burnout in the medium run, and it systematically misdirects effort toward the avoidance of self-blame rather than toward the actual goal. A person driven by guilt will do the minimum that suppresses the guilt feeling, which is typically not the same as doing the maximum that serves the goal.

The should audit. One of the series' concrete techniques is the experimental removal of all sense of obligation: notice what you would still care about if no one expected anything of you, if no standard demanded compliance, if the only driver were genuine interest and care. What survives this removal is the intrinsic value, the thing that can sustain effort without punishment. The technique is diagnostic as well as prescriptive: it reveals how much of one's ostensible moral seriousness is really about meeting a standard and how little is actually about caring for the thing the standard was meant to track.

The alignment parallel. Soares himself draws the connection explicitly: the difficulty of getting a mind's actions to reliably track its genuine values is not unique to artificial systems. It is a general feature of minds shaped by optimization processes, biological and cultural, that do not transmit their objectives into the motivations of what they optimize. Humans are the proof of concept: evolution optimized us for reproductive fitness and produced creatures who want many things that diverge from it, including guilt as a social-compliance mechanism that was adaptive in small groups and now misfires in the context of personal commitments to civilization-scale problems.