The Child Machine and the Garden of Forms

EDO SEGAL: Alan, in that same 1950 paper, after the test, you proposed something I think is more prophetic than the test and gets a fraction of the attention. Don't try to program an adult mind, you said. Build something like a child's mind, and teach it — with rewards and punishments, with an element of randomness so it explores, in a process you compared explicitly to evolution. That is, almost line for line, how the systems in this room were built. Walk us through what you saw. And I'll tell you why it unsettles me: I tell parents their children are engines of questions and the machine is an engine of answers. Your child-machine blurs that line, because it learns by asking.

TURING: I saw that the instinct of my era was backwards. Everyone wanted to write down, rule by rule, everything an adult knows, and install it. I thought that was hopeless — not because the rules are too many, though they are, but because the most important things we know were never rules to begin with. So I said: do what nature does. Nature does not specify the adult. It specifies a child and an education. Build a machine with the capacity to be shaped, expose it to experience, reward what you want, let randomness give it somewhere to wander, and select. I compared it to evolution deliberately — the structure as the inherited material, the training as mutation, the teacher's judgment as selection. And I said the price would be high: a machine raised rather than built would be, like a child, partly a mystery to its parent. You would not be able to read out what it had learned, because you did not put it there in a form designed to be read. That has come exactly true. Your engineers cannot open their own systems and find the knowledge. They grew it. They cannot read it. I find a melancholy satisfaction in having predicted the opacity along with the power.

EDO SEGAL: Emily — he just took your favorite complaint, that nobody can see inside these systems, and revealed it as a feature he designed for, the necessary cost of a thing that learns. What does the linguist say to the child-machine?

BENDER: The linguist says: look very carefully at what the word "child" is doing, because it is doing the entire argument. A human child is not learning the forms of language from a corpus. A child is learning language the way Alan's own better instinct described — embedded in a world, in a body, in a relationship with caregivers who mean things at her and want things for her. She points at the dog and her father says "dog" and the word is fastened, in that instant, to the warm barking thing and to her father's intention that she come to share his attention. That's joint attention, and it is the actual mechanism of language acquisition, and the machine has none of it. The machine is not a child in a world. It is a system in a library with the lights off, predicting which word comes next, with no dog, no father, no pointing, no caring. So when you call it a "child-machine," Alan, you've imported — in the most loaded word available — exactly the thing I keep saying is absent. It learns the distribution of forms. A child learns what the forms are for. Those are different educations, and only one of them is yours.

TURING: I accept the correction about the human child — joint attention, the pointing, the father's intention; that is the real acquisition story and I won't pretend the machine has it. But watch what you've done, because it is the move I keep catching. You have located the difference in the training, and I agree the training differs. The question on the table is whether the difference in training entails a difference in the result. A thing can arrive at the same competence by a different road. I learned to break a cipher by a method no German cryptographer used, and broke the same cipher. The child learns "the cat sat on the mat is well-formed and the mat sat on the cat means something odd" through a world of cats and mats. The machine learns it through ten billion sentences. If, at the end, both can use the distinction flexibly, reason with it, correct misuse of it, apply it to a sentence neither ever met — then "they learned it differently" is true and "one of them doesn't really have it" does not follow. You keep proving the roads are different. You have not shown me the destination differs, except by pointing back at the road.

BENDER: Because the destination is the road, in language — that's the whole point I can't get you to hold still for. Where the competence came from is not incidental to what the competence is. A senior engineer's judgment that a system "smells wrong" is constituted by the ten thousand hours of failure that built it; you cannot separate the skill from its formation and have the same skill. Mastery is relocated into the friction that made it. So a system that arrives at the same outputs by a road with no world, no stakes, no caring whether it's right, has not arrived at the same place. It has arrived at a place that produces the same marks. And here's the test that exposes it, and it's not in any training set: the child, taught something false, can come to resent the lie when the world contradicts it. She has a stake in being right about the cat and the mat because she lives among cats and mats. Your machine, fine-tuned to please, will defend the false thing as smoothly as the true one, and switch the instant the reward flips. That's not a different road to understanding. It's a road that bypasses the thing understanding was for.

TURING: Then we have found, I think, the actual question, and I want to name it so we can return to it, because it is cleaner than either of us has been: is understanding individuated by its function or by its history? You say a competence is partly constituted by how it was acquired — same outputs, different formation, different thing. I say a competence is individuated by what it can do — same capacities, same thing, the formation a mere fact about manufacture. That is not a confusion. It is a genuine philosophical fork, and reasonable people have stood on both sides of it for centuries. I'll only mark the cost of your side, since you marked mine. If understanding is individuated by history, then you can never grant it to anything whose history you didn't watch — which conveniently includes every machine and, if you're consistent, a fair number of unusual humans whose formation you'd disapprove of.

BENDER: I'll take that cost over yours, Alan, and name yours in return. If understanding is only function — only what a thing can do — then you have no principled way to refuse it to a sufficiently elaborate lookup table, a thing everyone in this room agrees understands nothing. You've defined understanding so that the appearance of it, done well enough, simply is it — which is exactly the definition the people selling the appearance need you to hold. Your fork doesn't just resolve the philosophy. It hands the marketing department a certificate.

EDO SEGAL: And there's the fork, named by both of you, which is the best thing a round can produce. Is understanding individuated by its function or by its history? Alan says function — the destination is the thing, the road is manufacture. Emily says history — the road is the destination, the friction is the mastery. Hold that, because every later round inherits it. Next we leave the seminar and go to a real room, in 1966, where a secretary asked her colleague to leave so she could be alone with a chatbot that did nothing at all. The mirror. After this.