The Mill, the Dark, and the Dopamine

EDO SEGAL: Gottfried, in the seventeenth section of the Monadology you wrote what I think is the single most prescient passage in the philosophy of this whole subject. Tell it the way you'd tell it to a sharp fifteen-year-old. And then, Rich — before you take it apart, I want you to do something unusual. Steelman it. Tell us what the mill gets right.

LEIBNIZ: Gladly, for it is the argument I am most sure of and it has only grown teeth with age. Imagine a machine built so as to think and perceive and feel — built to produce all the phenomena of a mind. Now imagine we enlarge it, keeping the proportions, until it is big enough to walk inside, the way one walks into a mill. We go in. And what do we find? Parts that push one another. Levers, gears, signals, one thing moving the next. Nowhere among the pushing parts do we find a perception — nothing that is about anything, nothing for which the motion means. Mechanism, I concluded, can produce only mechanism; perception cannot be explained by figures and motions. Now hear the modern sting, which I could not have known and which delights and horrifies me: you have built the mill I imagined, and unlike any brain it is open to inspection. You can walk inside. You can read every weight, trace every multiplication, watch every signal cross every layer. And what do you find? Exactly what I said you would find. Numbers pushing numbers. Arithmetic of unfathomable extent, and nowhere a meaning, nowhere an understanding, nowhere anyone at home. Three hundred years I waited to be checked, and the machine checked me, and I was right.

EDO SEGAL: Rich. Steelman first.

SUTTON: I can do that honestly, because the mill gets something deeply right that my whole field needs tattooed somewhere it'll see every morning. You can have all the mechanism and still not have found the thing you were looking for. We open these networks up, we read the weights, and we cannot find the understanding — that's true, that's the interpretability crisis, and it's real, and Gottfried called it in 1714. He's right that inspecting the parts doesn't reveal the meaning. He's right that mechanism is, on its face, just pushing. And he's right that this should bother us — that we've built something whose competence we can't locate in its components. That's the honest steelman, and it's a good argument. Now —

LEIBNIZ: Now comes the "but." I have waited three centuries for this "but."

SUTTON: [laughter] — now comes the but, and it's the same but I give everyone. You walked into the mill and didn't find a perception, and concluded perception can't be made of parts. But walk into a brain, Gottfried. The same mill. Three pounds of meat in a dark box, neurons pushing neurons, ions across membranes, one signal triggering the next — pure mechanism, parts pushing parts, and if you go looking among them for a perception you will not find one there either. There's no meaning visible in a synapse. And yet you don't doubt that you perceive. So the mill argument proves too much: it proves brains can't perceive, which we know is false. The conclusion isn't "mechanism can't make minds." The conclusion is "you can't find a mind by looking at the parts" — which is a fact about looking, about the level you're inspecting at, not a fact about whether the mind is there. Meaning isn't in the parts in your machine or in mine. It's in what the whole system does — in how it predicts, acts, and corrects against a world. You looked for understanding in the gears. It was never going to be in the gears. It's in the relationship between the gears and the world, and that you can't see by walking inside.

LEIBNIZ: You have given my own resolution back to me in materialist dress, and I must acknowledge it, because I made the same move you just made — I simply made it toward spirit and you make it toward the world. I, too, said perception is not in the parts. That is why I concluded the things that truly perceive are not machines at all but simple, indivisible substances, monads, each carrying its perceptions by its own nature. You cannot follow me there; the monad is not a hypothesis your age can use. But see what we share: we both deny that the perception sits among the pushing parts. You relocate it into the system's commerce with the world. I relocated it into a simple substance. Both of us fled the same emptiness. The difference is that I do not believe your relocation succeeds. "The relationship between the gears and the world" — where is that, sir, if not also a set of pushing parts, signals in and signals out, more mechanism? You have not escaped the mill. You have drawn a larger mill around the first one and called the air between them meaning.

EDO SEGAL: That's a real blow, Rich. He's saying: you put the meaning in the loop between agent and world, but the loop is just more mechanism — more parts pushing — so you've only made the mill bigger, not found the ghost. What's your answer?

SUTTON: My answer is that I don't need a ghost, and the reason I don't is the most surprising thing I learned in fifty years, and it isn't philosophy — it's a fact about brains. In the 1990s, neuroscientists looked at the dopamine system, the thing that drives learning in living animals, and found that certain neurons compute exactly the quantity at the center of my theory. Not reward itself — the difference between expected and received reward, the temporal-difference error, the signal in my 1988 equations. The math I derived from pure computational considerations turned out to be the math the brain runs in wet tissue. Now sit with that, Gottfried. The thing that makes a rat learn, and makes you learn, and makes my machine learn, is the same computation. So when you ask "where is the meaning, if not in pushing parts" — my answer is that meaning is what a particular kind of pushing does: a system that predicts the consequences of its actions, is surprised when reality differs, and changes itself to be less surprised next time. That loop, run long enough against a world that pushes back, is what knowing is. It's not a ghost added to the mechanism. It's a property of one specific mechanism — the learning one — that your brass gears, which only multiplied, never had. Your mill couldn't learn. That's why it was empty. Mine learns. That's the whole difference, and it's not nothing — it's everything.

LEIBNIZ: [long pause] The dopamine I did not know, and it is the strongest card you have played. That nature also computes the difference between expectation and outcome — that learning has the same shape in the rat and in the machine — this is the kind of convergence I most respect, where a thing derived from pure reason is found written in flesh. I held that the rational order of things would be mirrored in nature; you have handed me an instance and I will not pretend it does not move me. But I will hold my ground on the last inch, because it is the inch the whole evening is about. You say your loop learns and my mill did not, and that learning is the difference between empty and occupied. I grant the difference in capability entirely. I deny that you have shown it makes the difference in perception — that there is now someone home in the learning loop rather than merely a more elaborate emptiness, a mill that rebuilds itself but is still, all the way down, parts that push. You have shown me a mechanism that improves. You have not shown me that it feels its improving. And that — whether there is anyone there to whom the knowing happens — is the one thing neither your dopamine nor my monad has settled, and I suspect it is the one thing we will leave this room still holding.

EDO SEGAL: Hold that exact thread — is anyone home in the learning — because it comes back in the last hour and it will be sharper then. But we've been high up, in mills and monads, and I promised the reader the kitchen table. The next round comes down the stairs to a real person learning a real thing the hard way — to mastery, and the apprentice, and whether a machine that learns everything from experience quietly steals the experience the rest of us needed to learn from.