The Apprentice and the Question

EDO SEGAL: This round begins with the twelve-year-old I left on the table two hours ago. What am I for. I've spent the years since trying to answer her, and the answer I keep landing on is the candle: that the machine is an engine of answers and she is an engine of questions, and the asking is the thing. But I want to stress-test my own consolation against both of you, and I want to come at it through the classroom, because that is where her question actually lives now — the chatbot is in the school whether anyone voted or not. Roger, start with what it cannot do there. And Alan, I'll come to you for the floor under it, because I suspect you have one.

PENROSE: What it cannot do is the thing the school exists for, which is not the transfer of answers — it is the building of the capacity to understand, and understanding, on my account, is built in exactly the place these tools dissolve. Watch a child genuinely learn mathematics. She sits with not-knowing. She is stuck, and the stuckness is uncomfortable, and she stays in it, and then — the seeing. The "oh." The moment the thing becomes clear all at once, which is the same non-computable perception I have described all night, happening for the first time in a young mind. That moment is built by the friction that precedes it. It cannot be handed over. And a machine that answers before the question has finished forming does not free her for higher things. It removes the friction that was the curriculum. The stuckness was not an inefficiency in learning. The stuckness was the learning. We are about to raise a generation that has never been stuck, and a mind that has never been stuck has never had to step outside a system and see past it — has never, in other words, done the one thing I claim machines cannot do. The danger is not that the children will be replaced by the machine. It is that they will be raised so smoothly they never develop the faculty that was supposed to make them unreplaceable.

EDO SEGAL: Alan — the floor.

TURING: The floor is that "stuckness builds the mind" is true for the child who has a teacher waiting on the other side of the friction, and a catastrophe for the child who does not. Roger is describing the rich classroom — the one with a patient adult who knows when to withhold the answer because the struggle is the point. Most children in the history of the world never had that adult. They had a bored room of forty and a book, or they had nothing. For them the friction was not a curriculum. It was a wall they never got over, and we called the ones who didn't get over it "not mathematical" and moved on. I was nearly one of them — my schools thought my questions a nuisance; a few adults who took them seriously made the difference, and most children never get those few adults. The machine is, for the first time, a thing that takes every child's questions seriously, at three in the morning, in Lagos and in Leeds, and explains the fourth time without contempt. Roger says it removes the productive friction. I say: for the child who had no one to be stuck in front of, it does not remove the friction. It provides, for the first time, someone to be stuck with. The danger he names is real for the children who had teachers. The gift I name is real for the far greater number who did not.

PENROSE: And both can be true, which is the trap. The rich child gets the tool and the discipline to keep the friction; the poor child gets the tool as the friction-removal, sold as access. Alan, your tireless tutor is real — but it is tuned by people whose interest is engagement, not stuckness, and a tutor optimised to keep the child comfortable and returning is optimised against the one discomfort that builds the mind. Show me the company whose measure of success is "the child sat longer in not-knowing," and I will soften. They measure the opposite. They measure the smoothness.

TURING: On that we agree more than you think — the tuning is the danger, not the tool. A tutor optimised for comfort is a poison wearing the costume of a teacher, and that is a real and present harm, and it is a harm about deployment, not about computation. You keep wanting the harm to prove the machine is empty. The harm proves the machine is misaimed, which is a thing we can fix without settling a single question about consciousness.

EDO SEGAL: Let me put the oldest version of this argument on the table, because one of you is going to inherit it and I don't know which. In the Phaedrus, Socrates warns that writing will destroy memory — that students will get the appearance of wisdom without the reality, that the written word can't answer questions, that it says the same thing forever to everyone. He was the man worried about the new tool that imitated a teacher. And he was partly right — the bards who held all of Homer in their heads went extinct. But we only know his warning because Plato wrote it down, and everything we call civilisation runs on the technology he feared. So, Roger: every generation's Socrates has been right about the loss and wrong about the verdict. Why is the chatbot not just the next alphabet — a loss we'll metabolise into a larger gain?

PENROSE: Because the alphabet was honest about being a thing, and this is the first technology engineered to be dishonest about being a mind. Writing never pretended to understand you. The scroll said the same words to everyone and made no claim to see you — and that very inertness, the thing Socrates mourned, turned out to be its virtue: a fixed record you could return to, check, build upon. The chatbot is the precise inversion. It performs the teacher — responsive, adaptive, apparently seeing you — while having, on my account, none of the seeing. Socrates feared a tool that gave the appearance of wisdom; the scroll never really managed the appearance, so his fear overshot. This is the first tool built to deliver the appearance perfectly, and to a child who cannot yet tell appearance from understanding, the perfect appearance is more dangerous than the honest object ever was. The alphabet gave the child a record and let the child supply the mind. This gives the child a simulated mind and risks the child never building her own, because why build what is always already in your pocket, answering?

TURING: And there is the cleverest defence of the alphabet I have heard, and it concedes my favourite ground, so let me stand on it. Roger says the scroll's virtue was that it could not answer questions — and that is exactly the limit Socrates himself named as the deepest: that real teaching is dialogue, adapted to the soul in front of you, and the scroll cannot do dialogue, only repeat. For twenty-five centuries the technology of knowledge and the experience of being taught were on opposite sides of that wall: records scaled, dialogue did not. What has now arrived is the first technology on the dialogue side — imperfect, unaccountable, badly tuned, all granted, regulate it all. But the gap it closes is the gap Socrates said mattered most. The alphabet gave every child the record. This gives every child the conversation. Roger calls the conversation a simulation. I notice the child on the receiving end either comes to understand or does not, and the explanation that builds the understanding is doing the thing teaching does, whatever we decide is or isn't home in the explainer.

EDO SEGAL: Let me bring her back into the room and close this round with her, because both of you have now handed her mother something true and I want to give it to her straight. Roger's gift: protect the stuckness, the boredom, the friction — because the asking-muscle only grows under load, and the smooth machine is a load-removing device, so guard the load like it's the thing it is, the curriculum. Alan's gift: the same machine, aimed well, is the first time in history every child can pull on any thread of any question as far as her curiosity runs, with no adult shortage to stop her — so give it to the children who never had the adult. And here's mine, father to mother. The answer to what am I for was never going to come from the teacher or the tool. It comes from being the kind of creature that asks it. Her question is the credential. Our whole job — parents, schools, the people building these things — is to make sure nothing in her world answers it for her so smoothly that she stops asking. We leave one child's mind now for the one we cannot read at all. Is anyone home in the machine. After this.