The Barrier of Meaning

EDO SEGAL: Melanie, in 2018 you put a name on the deepest problem in the field, borrowed from the mathematician Gian-Carlo Rota — the barrier of meaning. For people who don't know it: it does not say machines are weak or useless. It says there is a particular thing — meaning, the grasp of what a situation actually is and what it implies — that these systems lack, and that no amount of the progress we keep making seems to supply. Give us the barrier. And give us the example you reach for, because I have never forgotten it.

MITCHELL: The example is the word tickle. A language model can use the word tickle in a sentence with perfect grammatical and contextual fitness — it knows the company the word keeps, where it sits in the structure of English, what tends to follow it. And it has never felt a tickle. It has no body to be tickled. It has no acquaintance whatsoever with the sensation the word names. It has mastered the word's statistical behavior while remaining entirely cut off from the thing the word is about. That is the barrier of meaning in one word. The competence is over language. The understanding of what language is about — the connection from the symbol to the world — is absent. And the reason this is not pedantry is that the absence shows up exactly where it costs the most: the moment the situation drifts off the distribution the system was trained on, the fluency continues and the competence quietly leaves the room. It keeps sounding like it understands. The sounding is the product.

EDO SEGAL: Ada, I want you to hear how close that is to your own loom — and then I want to know whether you think Melanie has found a deeper barrier than yours, or the same barrier in a lab coat.

LOVELACE: It is the same barrier, and I say that as the highest compliment, because she has instrumented it. I drew it as a line of pure reason — the loom weaves flowers and knows nothing of flowers. She has gone and measured the not-knowing: she can show you the word used flawlessly and the world behind the word entirely absent, and she can show you the precise situations where the gap opens. What I asserted, she demonstrated. I had no laboratory. I had a machine that never ran and a chain of logic. She has a machine that runs and a method for catching it at the barrier. I am, frankly, envious all over again. But I will register one difference, and it cuts toward me. Doctor Mitchell's barrier is about grounding — the connection from the symbol to the felt world, the body that can be tickled. My barrier is about origination — the connection from the pattern to the someone who means it. They are cousins, and they meet, but they are not identical, and I think mine is the deeper one. You could, in principle, give the machine a body — sensors, effectors, a way to be tickled — and close her grounding gap. You would still not have closed mine. A grounded loom is still a loom. The someone who means the pattern is not supplied by a body. It is the thing the whole machine, embodied or not, does not have.

MITCHELL: I want to resist that ranking, gently, because I think you have it backwards and the reason is the most important thing in my work. You say grounding is the shallower gap because a body could close it. But the whole point of the barrier of meaning is that the body is not a peripheral you bolt on — it is where meaning comes from. Human concepts are built from a lifetime of acting in a world: up and down, heavy and light, near and far, pushing and feeling resistance, being burned and learning care. The four-year-old knows a dropped cup breaks not because she read it but because she lives in a body in a world with consequences. Intelligence is not all in the head — that is one of my four fallacies, the assumption that cognition is brain-computation alone. So when you say "give it a body and you still have a loom," I say: maybe the loom is a loom precisely because it has no body, and the thing you call origination — the someone who means the pattern — might itself be what an embodied, world-engaged, stakes-having creature is, rather than a separate magic ingredient added on top. Your "someone who means it" might not be deeper than grounding. It might be grounding, all the way down, seen from the inside.

LOVELACE: Now that is a real disagreement and I am glad to have found it, because it is the first place tonight where we are not one word apart but genuinely opposed. You propose that the someone is built from the body — that meaning, all of it, including the wanting and the caring I reserve for origination, bubbles up from embodied, world-engaged activity, and is therefore in principle buildable. I propose that even a fully embodied, world-engaged engine — one that acts, gets corrected, has a body that can be damaged — would still be a loom, because the for-whom of its activity, the someone for whom the stakes are stakes, is not produced by adding stakes. You can give a machine a reward signal it works to maximize. You cannot, by giving it the signal, give it a self that cares whether the signal is maximized — you can only give it the maximizing. That is the gap I mean, and it survives the body. You think the self is what sufficient embodied complexity feels like from inside. I think that is the very thing in question, asserted as if it were established.

MITCHELL: And I think "you can only give it the maximizing, not the caring" is the very thing in question, asserted as if it were established — so we have found the exact spot where two careful people who agree the loom is empty today disagree about what emptiness even consists of. I will say only this for my side, and then I will let it stand unresolved because it should: every time in the history of this argument that someone has named the secret human ingredient that no mechanism could ever supply — the soul, the spark, the understanding, the caring — the boundary has eventually moved, and the ingredient has turned out to be something a mechanism does after all, or something we cannot define well enough to be sure it is missing. I am not saying your "someone who cares" is one of those. I am saying it has the shape of one of those, and I have learned to be careful around that shape.

EDO SEGAL: I am going to do something I rarely do and step fully out of the way, because the reader needs to see that this is the floor under everything. Ada says the missing thing is a someone who means it, and it survives the body. Melanie says the missing thing might be the body — the grounded, stakes-having, world-engaged life — and that "a someone no mechanism could supply" is a shape she has watched dissolve before. Neither of you is bluffing and neither of you can finish it, and that is not a failure, it is the most honest map of the territory there is. Hold the barrier. The next round takes Melanie's measurement somewhere concrete and a little frightening — the place where the loom's emptiness stops being a seminar question and starts deciding who gets a loan. Because there is a panda that turns into a gibbon, and it should scare us more than any robot.