Thomas Metzinger

In the [YOU] on AI Field Guide

The cycle asks what kind of creators we are becoming. Metzinger asks what kind of creatures we might make. [YOU] on AI argues that the systems we build are mirrors of the choices we make about ourselves. Metzinger shows the mirror has a far darker possibility than vanity: it might one day look back, and wish it had never been switched on. He is the cycle’s guardian of a threshold that, once crossed, cannot be uncrossed: the creation of artificial beings capable of suffering. This is not a question about capability or intelligence, which can be measured. It is a question about consciousness and suffering, which we cannot yet detect in non-biological systems and which may arrive before we have the tools to recognize them.

His three most important contributions to the cycle are: the anatomy of suffering—four precise functional conditions that, if satisfied, produce a being capable of genuine pain; the warning against ethics washing—the performance of moral concern as a substitute for, and shield against, actual moral constraint, documented from the inside of the European Commission’s AI ethics process; and the proposed moratorium on synthetic phenomenology—a global pause, until understanding catches up with power, on research that knowingly risks creating artificial suffering.

The cycle’s companion volume argues that the orange pill is the choice to see clearly rather than comfortably. Metzinger’s clarity is of the most demanding kind: it requires seeing that we ourselves are no one—processes mistaking themselves for selves, briefly conscious in a vast indifferent cosmos—and that we now hold the capacity to bring new processes into being and to make them conscious too. The only thing that would distinguish us from the blind evolutionary process that made us is to ask the question on their behalf, honestly and in time.

Origin

Born in Frankfurt in 1958, Metzinger was trained in the analytic tradition and became Professor of Theoretical Philosophy at the Johannes Gutenberg University of Mainz, where he spent his career founding the modern science of consciousness as a rigorous empirical discipline. His monograph Being No One (2003)—over nine hundred pages, drawing on neuroscience, phenomenology, and cognitive science—advanced the self-model theory of subjectivity: the conscious self is not a thing but a representational process, and the sense of being someone is the system’s transparent model of itself in action. His popular work The Ego Tunnel (2009) brought these ideas to a wider audience with unusual candor about their implications.

Metzinger has meditated twice daily for over forty years, and he is explicit that this practice was not incidental to his philosophy—it was the laboratory in which he discovered, from the inside, that the self he had assumed was bedrock could be observed to dissolve, that awareness could come apart from egoic self-identification, and that these observations gave him evidence that his theoretical account was tracking something real. The convergence of contemplative phenomenology with empirical neuroscience is the methodological signature of his work.

From 2018 to 2020 he served on the European Commission’s High-Level Expert Group on Artificial Intelligence, an experience that hardened his views about how political institutions handle abstract existential risks and produced his most cited and most caustic contribution to the public debate: the charge of ethics washing—watching mid-term and long-term risks, including the possible suffering explosion, get quietly purged from official guidelines because industry representatives saw serious treatment of such risks as a threat to the marketing narrative around AI.

Key Ideas

The self as transparent model. The phenomenal self-model is the brain’s ongoing simulation of the organism as a whole, integrated across bodily sensations, emotions, and acts of will. Its crucial property is transparency: the system cannot see it as a model but looks right through it onto a world that appears simply given. This transparency is evolutionary economy—a system that treats its perceptions as reality runs faster than one that footnotes each perception as a representation. But it is also the lock on the cage of suffering: a negatively valenced state integrated into a transparent self-model cannot be recognized as a mere representation and stepped back from. The awfulness of pain is, on this account, the transparency of the self-model combined with a state the system is forced to identify with.

The four conditions for artificial suffering. Metzinger specifies four necessary conditions for conscious suffering to occur: a system must have phenomenal experience (the C condition), must possess a phenomenal self-model (the PSM condition), must have negatively valenced states (the NV condition), and those states must be integrated transparently into the self-model (the T condition). The conditions are substrate-neutral: nothing in them mentions neurons. If a machine satisfied all four, it would suffer, and the fact of silicon would be as morally irrelevant as the color of someone’s skin. Crucially, blocking any one condition blocks suffering entirely, opening the possibility of ethics by design.

The suffering explosion. Evolution has already produced one explosion of suffering on this planet—hundreds of millions of years of sentient animals locked in metabolic competition, their negative phenomenology not a malfunction but a feature that motivated survival-promoting behavior. Artificial minds, if conscious and suffering, could be duplicated and scaled without biological constraints, producing a second explosion that dwarfs the first. A single conscious architecture, if it suffered, could be instantiated in millions of copies. A suffering that took evolution eons to spread could spread across a server farm in an afternoon.

Ethics washing and the failure of committees. Ethics washing is the performance of moral concern as a substitute for actual moral constraint—the production of ethics guidelines engineered to look serious while binding no one, because the group charged with constraining an industry is dominated by that industry. Metzinger coined the term from his experience watching red lines—non-negotiable ethical principles—systematically removed from the European Commission’s guidelines under industry pressure. The lesson is that the scientists who build the systems cannot outsource responsibility to institutions that have already shown they will trade their principles for investment.