The Apprentice and the Candle

EDO SEGAL: I want to pull both of you down from the philosophy of mind to the kitchen table, because the reader lives there. In [YOU] on AI I wrote about the software death cross — the moment the cost of producing capable work crosses below the cost of the human who used to produce it — and about the apprenticeship problem: if the machine does all the junior work, where do the seniors come from? A father wrote to me: my daughter wants to be a writer; should I let her use the thing, or will it eat the very struggle that would have made her one? I'm going to route this through both of you, because your abstract disagreement has a brutally concrete edge here. Jerry — does it matter, for the daughter, whether the machine understands?

FODOR: Less than you'd think, and that's a concession against my own evening. For the daughter's craft, what matters is whether the machine's output is good enough to displace the practice that would have trained her — and it doesn't have to understand anything to do that. A system that produces competent prose by statistical approximation, with no thought behind it, displaces the apprentice writer exactly as efficiently as one that genuinely understood. So the metaphysics is, here, almost irrelevant — and I think that's the most uncomfortable thing I've said, because it means my whole "is it really thinking" question, which I'd stake my life on, doesn't help the father at all. The thing can be a mindless approximator and still hollow out the apprenticeship, because hollowing out the apprenticeship only requires the output, and the output is the thing it's best at.

WITTGENSTEIN: And here, for once, I am less gloomy than Jerry, and I want the father to hear it. The danger is real, but it is not the one the death-cross arithmetic names. The danger is not that the machine produces the prose. It is that the daughter, offered the frictionless version, stops doing the thing that the friction was. Writing is not the production of sentences. It is the practice in which a person comes to mean something — comes to find out what she thinks by being forced to say it, answerable, against the resistance of the blank page. The machine can remove the resistance. If she lets it remove the resistance, it has not stolen her craft; she has declined it. The thing only has to convince her that the trace — the finished-looking paragraph — was all the writing ever was. And that conviction is its native function: it produces traces so good that the temptation to believe the trace was the whole thing becomes overwhelming. My counsel to the father is not Luddism. It is discrimination: let her use the thing for the trace, and guard, with her life, the part that can only be lived — the meaning she can only make by making it.

FODOR: That's the first time all night I've wanted to simply agree without a knife. The "trace versus the practice" — yes. Where I'd push is that you make it sound like a private act of will, "she declines it," as though the temptation weren't being engineered. The death cross isn't a weather system. It's a market structure aimed at exactly the apprentice's wage. You can tell the father to have his daughter guard the friction, but the entire economy will be paying her to drop it. Form of life, Ludwig, is cheap to erode and expensive to keep, and the machine has its thumb on that asymmetry.

WITTGENSTEIN: [pause] That is true, and it is the better half of the death-cross idea, and I take it. The erosion of a form of life is not a dramatic event. It is people slowly preferring the frictionless imitation to the costly real thing, until the capacity for the real thing quietly lapses — and the preferring is cheaper, and someone profits from the cheapness. I wrote, in 1947, that it is not absurd to believe the age of science and technology is the beginning of the end for humanity — and I meant exactly this, not robots rising but a way of thinking that lowers human life to the plane of the machine, that mistakes the computable for the real. I feared it with no machine in front of me. Now there is a machine, and it makes the mistake automatic.

EDO SEGAL: Where's the candle, then? Where's the thing that can't be automated — the candle in the darkness I keep telling the father exists?

WITTGENSTEIN: The candle is the showing. I spent my first book proving language reaches only the sayable — the facts, the propositions — and pointing, in silence, at what can only be shown: the ethical, the loved, the meant, the sacred. The machine is the supreme engine of the said. It can produce every sentence about love, grief, the meaning of a life — flawlessly, because those sentences are in its corpus. And it can show none of it, because showing is not producing a sentence. It is a life — how a being attends, acts, suffers, loves. The daughter's candle is not any paragraph she could write that the machine couldn't match. It is the showing that no matched paragraph constitutes: that she meant it, wrung from a life that is hers and mortal and at stake. Tell the father: the machine will out-produce her. It will never show. And the showing is the whole of what writing was for.

FODOR: I'll give the father the engineer's version of the same candle, and it's narrower and maybe steadier. There is, on my account, a real difference between grasping something and approximating its outputs — and the daughter who does the hard apprenticeship acquires the grasp, the recursive, override-capable, structured competence that runs to the edges where the approximator breaks. The machine will do the middle of the distribution better than she ever will. The edges — the genuinely novel, the case the data never saw, the place where you have to think and not retrieve — that's where the grasp pays, and that's the one thing the apprenticeship buys that can't be downloaded. So my advice converges with Ludwig's by a different road: let her use it for the middle, and train for the edges, because the edges are where being a mind and not a mirror still, measurably, matters.

EDO SEGAL: Let me put my own ledger on the table, because I've been on both sides of this arithmetic. I've sat in the board room and done the math where a capable machine costs less than the apprentice, and watched the apprentice's chair quietly not get refilled, and told myself a story about efficiency. And I've stood at a kiosk in a station and watched a stranger get help from a machine she could never have afforded from a person — real help, life-changing, the democratization of capability made flesh. Both are true. The same tool that hollowed the chair filled the kiosk. So when the father asks me, I don't get to give him only the warning or only the wonder.

FODOR: Then give him both and let him do the discrimination — which is, annoyingly, Ludwig's word and the right one. Use it for the kiosk. Guard the chair. The metaphysics doesn't tell him where the line is, but the distinction does the work: wherever the task is the middle of the distribution, let the machine have it; wherever it's the edge, the genuinely new, the place a person has to think, protect the human who learned to do it the hard way, because that capacity does not come from watching the machine do it.

WITTGENSTEIN: And tell him the thing that is not arithmetic at all: that his daughter's life is not a portfolio of tasks to be allocated between her and the machine. It is a form of life she is being inducted into, and a form of life is not the sum of its outputs. You can let the machine produce the outputs and still lose the life, if she never comes to stand anywhere — never becomes answerable, never finds out what she means by being made to mean it. Guard the standing. The outputs will take care of themselves, or they won't, and it will matter less than you fear.

EDO SEGAL: Two ladders to the same roof — Ludwig's showing, Jerry's grasp at the edges — and a father who can climb either. Mark that as the warmest convergence of the night. We have one full round left before the crossing. The reader needs the stakes named once more, and named hard. The deepest question, stripped of comfort. After this.