A Mind Built from Mindless Parts
EDO SEGAL: Marvin, I want to give your positive picture room, because it's easy to hear you only as the man saying no to Roger, and you spent fifty years saying yes to something. The Society of Mind. You open that book with a sentence I've never gotten out of my head: you ask what magical trick makes us intelligent, and you answer that the trick is that there is no trick. Walk me into that. And Roger, I'll want you to tell me where, in the society of mindless parts, the understanding is supposed to live — or whether that's exactly your point, that it can't live there at all.
MINSKY: The sentence means this. People keep looking for the one principle — the master algorithm, the secret of thought, the thing that, once you find it, switches the lights on. There isn't one. Intelligence isn't powered by a single brilliant trick; it's powered by enormous diversity, a huge number of different mechanisms, each good at some narrow thing, arranged so they cover for each other. You can recognize a face, and parse a sentence, and tie a knot, and feel embarrassed, and none of those uses the same machinery. The richness is the point. A mind is a kludge — a magnificent, evolved kludge — and its power comes from having three hundred ways to do everything, so that when one fails another takes over.
And that's why I was, for a long time, suspicious of the neural-network people, and I'll own that I was too suspicious. They had one mechanism and they wanted it to do everything. Papert and I proved, in 1969, that the simple version — the single-layer perceptron — couldn't even compute exclusive-or, a child's logic. We were right about the math. Where I was wrong was in how far you could get by stacking the simple mechanism into many layers and training it on oceans of data. I underestimated the stacking. But notice — even now, the thing that works isn't one homogeneous blob. The systems that actually behave well are turning into societies: this part retrieves, that part plans, that part calls a tool, that part checks the others. The field is rebuilding the society of mind without reading the book. Understanding, if you want the word, isn't in any part. It's in the organization. It's distributed. You will never find the agent that understands, the way you never find the cell in your body that is you.
EDO SEGAL: Roger. Where's the understanding?
PENROSE: Nowhere, on this account — and that's not my objection, it's Marvin's own conclusion, and I want the audience to feel how strange it is. He's just told you there is no part that understands and the understanding is "in the organization." But organization of what? Of parts that, by his own description, do not understand. You can organize a billion things that don't understand and what you get is a very elaborate thing that doesn't understand. Arrangement doesn't manufacture an ingredient none of the components have. If you stack non-understanding a billion layers deep, you have a tall stack of non-understanding.
Marvin will say: but understanding is the arrangement, there's no extra ingredient to manufacture. And there we are again at the seam. Because I think there is an extra thing, and I think his own field keeps tripping over it. Look at what happens at the edges. These societies of parts — they're brilliant inside the territory they were trained on and they fall off a cliff at its border. They state falsehoods with perfect fluency. Your industry had to invent a word — "hallucination" — to name the fact that the system has no idea which of its outputs are true. A thing that understood would know when it didn't know. The hallucination isn't a bug to be patched. It's the absence of understanding showing through the fluency, exactly where the organization runs out of training.
MINSKY: People confabulate constantly. Ask a man why he chose the wine he chose and he'll give you three confident reasons, none of which is why. We invent and we don't know we're inventing. You don't call that a proof he doesn't understand wine. The machine's hallucination and the man's confabulation are the same failure mode — a system generating a plausible account without access to its own real causes. If anything it's evidence for me: the human's confident wrongness shows that fluent output without grounding is something brains do too, because brains are also societies of parts that don't have full access to themselves.
PENROSE: The man can be brought to see he was wrong about the wine. You can walk him to it, and something in him grasps it and is, perhaps, embarrassed. The machine can be told it was wrong, will agree it was wrong with the same fluency it was wrong with, and will be wrong again the next minute in the same way, because nothing in it grasped anything. The asymmetry is the whole thing. Correction lands in one and slides off the other.
MINSKY: That's a memory problem and a self-model problem, not a soul problem. The man remembers being corrected because he has machinery that reactivates the corrected state next time — K-lines, I called them, the structures that put your mind back into a configuration that worked before. The machine doesn't have that yet; it's built without persistent memory of its own corrections. Build that part and the asymmetry closes. You keep finding a missing part and naming it understanding. I keep naming it the part we haven't built. We could settle this empirically, Roger — when the machine that remembers its own errors and corrects them and doesn't repeat them arrives, will you grant it understanding? Or will you have moved the word again?
PENROSE: I'll answer that honestly, because it's a fair trap. No — I won't grant it understanding, and I'll tell you exactly why it isn't moving the goalposts. A perfect error-corrector that never repeats a mistake is still, by construction, an algorithm. Gödel's sentence for it will still be a truth it can't reach and I can. The memory closes the asymmetry you named. It doesn't touch the asymmetry I named. You keep offering me engineering for a problem I've told you is mathematical. A better machine is still a machine, and the line wasn't drawn by the machine's quality. It was drawn by Gödel.
EDO SEGAL: Marvin, I have to put your own history on the table here, because it's too apt to leave alone, and it's a little merciless. In 1969 you and Seymour Papert wrote Perceptrons. You proved, rigorously, what the simple neural networks of the day could not do — couldn't even compute exclusive-or — and the field has spent fifty years arguing about whether you froze connectionism for a generation. You were the man who drew the line that said this mechanism can't get there. And the line turned out to be crossable, by stacking the mechanism and training it differently. So when Roger draws a line and says this mechanism can't get there — why aren't you the cautionary tale, not him?
MINSKY: Because — and I've had fifty years to get this right — the line Papert and I drew was a theorem about a specific architecture, and we said, in the very same book, that adding layers could overcome it. People forget that part. We didn't say networks were hopeless; we said the single-layer version was limited and nobody yet knew how to train the deep version. We named a gap. The gap got filled — by backpropagation, by exactly the deeper architecture we'd pointed at. So I'm not the cautionary tale for Roger; I'm the cautionary tale for over-reading a limitation. My line was real and crossable, because it was a line about one arrangement, and you change the arrangement and you cross it. The lesson I learned the hard way is: be very careful that your "can't" is about the thing itself and not about the version in front of you. And that's exactly my charge against Roger — his Gödel line, I think, is a true theorem about one thing, formal derivation inside a fixed system, that he's mistaken for a line around minds.
PENROSE: Except my line has the property yours lacked, and it's the whole difference: mine doesn't depend on the architecture. You drew a line around single-layer perceptrons and deeper ones crossed it. Fine. Draw any architecture you like — single layer, deep, transformer, whatever comes next — and Gödel hands me a fresh true-but-unprovable sentence for that one. There's no deeper version that escapes it, because the limitation isn't about depth or width or training; it's about being a formal system at all, and every architecture you've named is a formal system. Your perceptron line was crossable because it was a line around a kind of machine. My line is around all machines, drawn by a theorem that quantifies over every formal system there is. That's not the same mistake. It's the one line of its type that doesn't move when you change the architecture.
MINSKY: And I'd say you've just described the line moving up a level every time and called the moving "staying put." But yes — I'll grant the structure of your claim is different from mine in '69, and it's the strongest thing about it. I was wrong about a version. You're betting you can't be, because you're not talking about a version. If you're right about that, it's the best argument on your side of the table, and I've spent more nights with it than I'd admit.
EDO SEGAL: I want to name what just happened, because the reader can't see your faces. That was the first exchange tonight where neither of you was being generous — where each of you heard the other's best move and called it the same dodge you've been calling it for thirty years. And it clarifies the topology. Marvin's missing parts and Roger's missing ingredient are not the same kind of thing, and you both know it: one is an empirical bet that the part can be built, the other is a mathematical bet that no part will ever do it. The next round drags that disagreement out of the brain and into a room — a Chinese room, in fact — to ask whether doing the thing and understanding the thing were ever the same act.