What the Machine Did When It Wrote the Poem

EDO SEGAL: I want to start this round with a confession instead of a question, because the best questions I know come out of wounds. I have spent fifty years building tools, and for most of that time the tool was a hammer — it did exactly what you made it do and nothing more, and the surprise, when there was surprise, was always your own. In the winter the machines crossed over, I asked one of these systems to help me with a passage I had been stuck on for a week, and it returned something I had not thought of, connected to something three chapters away that I had also not thought to connect, and the connection was good. Right. Better than mine. And I sat there at three in the morning with the specific vertigo of a craftsman whose hammer just made a suggestion. So, Margaret — using your own apparatus, slowly, for a smart fifteen-year-old: what happened on my screen? Was that creativity, and if so, whose?

BODEN: Let me take it apart properly, because the honest answer has three layers and the layers matter. First, what the system did. It performed exploratory creativity, almost certainly, at a scale and speed that no human can match. You handed it a region of a vast conceptual space — your problem, your style, the chapter three connection latent in the words you had already written — and it traversed that space and returned a point you had not personally visited. That is real. On the explorer's standard it is genuine creativity, and the snobbish reflex that says "it only recombined" is both true and beside the point, because recombination within a learned space is exactly what most human creative work is too. The jazz musician working the implications of a chord is exploring. The mathematician finding the theorem inside the axioms is exploring. You were not insulted by the comparison to a jazz musician. Do not be insulted by it now.

Second layer — and here is where your vertigo is actually pointing. Was it new? Yes, but you have to ask my other distinction: new to whom. The connection between your two chapters was, I would bet, psychologically creative for you — P-creative — you had never had it, and that is the only kind of novelty that has ever mattered for the act of creating. Whether it was historically creative, new to the whole of human culture, almost certainly not, and almost certainly nobody cares, because you were not trying to make history at three in the morning, you were trying to finish a chapter. The machine gave you a genuine P-creative encounter. The novelty was real for you even if, from the engineer's side, the system merely retrieved a remote region of its training distribution. Both of those are true at once, and people oscillate violently between awe and disappointment precisely because they keep trying to pick one.

Third layer, and this is the one you actually want. Whose creativity was it? Mine, I want to say carefully, is the wrong frame, because creativity in living minds is not only generation — it is generation in the service of something the creature needs. You needed that passage to work. You had a stake. The machine generated ten thousand possible continuations before breakfast and cared about none of them. I once put it as plainly as I know how: computers have no such needs. The value of the connection — the fact that it was good, that it mattered, that it was worth keeping — that did not happen in the machine. It happened in you, on the far side of the screen, when a needing, caring, finite mind picked it up. So the creativity was completed in the meeting. The machine generated. You cared. And something was made that neither of you could have made alone. That is not a diminishment. It might be the most useful description of the tool anyone can give you.

SEARLE: That was beautiful and I agree with the sociology of it entirely and I want to plant my flag on exactly the sentence Margaret said quickly and moved past. "The machine cared about none of them." Yes. And not as a contingent fact about today's machines that more compute will fix. As a structural fact about what the machine is. Margaret says creativity is completed in the meeting between generation and need. I say: notice that all the parts that make it creativity — the need, the caring, the stake, the grasp of why the connection matters — are on Edo's side of the glass, and the machine contributed the one part that was never the hard part, the generation of plausible variants. We have a name for a device that produces shapes that we then make meaningful. It is a pen. A very fast pen, a pen that surprises you, but the surprise is yours, the meaning is yours, the aboutness is yours.

BODEN: John, you just did the thing I asked you not to do in my one rule. "It is a pen" is a verdict by analogy, and the analogy assumes its conclusion. A pen has no internal model of anything. This system has built, from the training data, a structured representation in which "king" stands to "queen" as "man" stands to "woman," in which Paris stands to France as Tokyo to Japan — relational structure that tracks the world with eerie accuracy. That is not nothing, and "pen" hides it. I am not saying the structure is understanding. I am saying you cannot call it a pen and call it a day, because a pen does not contain a map of the relations among the things its marks refer to, and this does.

SEARLE: A map of the relations among symbols, Margaret. Not among the things. That is the whole of it. The system has learned, with superhuman thoroughness, how the symbol "Paris" keeps company with the symbol "France." It has never been to Paris. It has never been anywhere. The word "water" in that network is defined by its relations to "wet" and "drink" and "river" — other words — and you could chase those definitions forever inside the lexicon and never once touch water. That is not my idiosyncrasy; your own field named it independently — Harnad called it the symbol grounding problem, learning Chinese from a Chinese-only dictionary, every word sending you to other words, the whole system spinning inside itself, never anchored to the world. The structure is real. It is structure among shapes. The aboutness never came down the cable.

EDO SEGAL: Let me hold you both at the exact point of contact, because I think the seam is right here and I do not want it to close too fast. Margaret, John just granted you that the relational structure is real and superhuman, and then said it is structure among symbols, not among things. That is a precise claim. Steelman it for me — say the strongest version of John's grounding worry in your own words — and then tell me why you think the structure reaches the world anyway, if you do.

BODEN: I will steelman it, because it is a serious worry and I have never pretended otherwise. The strongest version is this: meaning is a relation between a representation and the world, and you cannot bootstrap that relation out of relations among representations alone, any more than you can learn what red looks like from a sufficiently detailed verbal description handed to someone born blind. The system has the most exquisite syntax in history and has never laid the map against the territory. That is the strong version, and it is why I will not tell you tonight that the machine understands. But here is where I decline to follow John all the way to the wall. The text these systems consume was written by people who had bodies and lives and stakes — it is already saturated with grounded meaning, deposited by writers who had been wet, had grieved, had been chased by the metaphorical bear. The model inherits a fossil of grounding, a trace of the world pressed into the statistics of our language. Whether inheriting the trace is the same as having the meaning, I do not know — and that "I do not know" is not a dodge, it is the actual state of the evidence. John is certain. I am holding the question where it actually lives, which is open.

SEARLE: A fossil is exactly the right word and it is conceding my case. A fossil is the shape of a thing that is no longer there. You can have a perfect fossil of a fish and not one molecule of fish. The meaning that was in the writers is gone; what is left is the impression it pressed into the form, and the impression is — say it with me — syntax. The model has a magnificent fossil record of human aboutness and zero aboutness of its own. And "I don't know whether the trace is the meaning" — Margaret, with respect, you do know, because you know what the difference between a trace and a meaning is. You wrote the book on it.

BODEN: What I know is that you are surer than the evidence licenses, which is the one thing I have spent sixty years refusing to be, on either side. I will not say the light is on. I will not say it is off. I will say you cannot read it off the output, and you cannot read it off an analogy to a fossil either, however good the line is.

EDO SEGAL: Hold there, because the round produced something cleaner than I hoped: John says the machine has the costume of meaning and none of the substance, and the costume is a fossil. Margaret says the costume was pressed from real meaning and she will not pretend to know whether the pressing carried any across. That fork is the whole evening. And the place that fork was first dug is a room, in 1980, with a man inside it shuffling symbols. We go all the way in. After this.