The Crossing

LEIBNIZ: Then I will ask you first, Mr. Sutton, the question I have held since your opening, and I will ask it as plainly as you would. You say we must not build knowledge in — that the learner must find its own. But you began the evening granting me a floor: an architecture under the experience, a form that makes learning possible. So tell me precisely: what is in that floor? Because if the floor is rich — if it contains the structure that makes some experiences learnable and others not — then I am right that knowing is composed, and we are merely arguing about where the composing happens, in the agent's lifetime or in its design. And if the floor is thin, nearly nothing, then you owe me an account of how a nearly-formless thing comes to carve the world so exquisitely. Which is it? A rich floor, and I win the principle? Or a thin floor, and you owe me a miracle?

SUTTON: Neither horn, and I think the dichotomy is the trap. The floor is thin but not formless — and the specific thing it contains is not knowledge, it's the capacity to learn and the drive to predict and a way of forming abstractions. That's not nothing, and it's not a miracle, and crucially it's not your kind of structure. Your structure is content — concepts, primitives, knowledge of the domain. My floor has no content at all. It's pure mechanism for acquiring content. So you don't win the principle, because the principle in dispute was never "is there any architecture" — of course there is — it was "is the knowledge built in or learned." And the knowledge is learned, all of it, because the floor contains none. You're trying to call the learning machinery "composed knowledge" so you can claim it. But a loom is not a tapestry. I'll grant you built the loom. The tapestry is woven by the world, and the tapestry is the only part that knows anything.

LEIBNIZ: A loom is not a tapestry — but a loom determines what tapestries are possible. You cannot weave a sphere on a flat loom. So the loom is not innocent of the cloth; it bounds it, shapes it, forbids some patterns and permits others. When you choose the agent's senses, its actions, its very capacity for abstraction, you have chosen the loom, and the loom has chosen the space of all tapestries it can ever weave. I do not need the content to be built in to win my principle. I need only that the space of possible knowings is fixed by the design — and you have just conceded it is. So the world weaves, yes. But it weaves only what your loom permits, and what your loom permits was composed, by you, in advance. The world is not free. It is free within your frame. That is composition wearing the mask of experience.

SUTTON: [long pause] — That's the best you've thrown all night, and I have to be careful, because there's something true in it and something I still think is wrong. True: the loom bounds the possible. A general learner with different senses, different actions, would learn a different world. Granted, fully. But here's the wrong part, and it's the part your whole worldview can't accept. The loom I'm describing is general enough that the bound doesn't bind in practice — it's like saying a Turing machine is "bounded" because it can only compute computable functions. Technically true, practically vacuous, because the space of the computable already contains everything we could ever want. My loom is the general learning loom: the bound it imposes is so wide that within it lies every knowing a situated agent could ever need — that's the difference between predicting what would happen and constructing what was specified, and it's the difference that makes the frame vanish in practice. So yes, there's a frame. But a frame that contains all achievable knowings is not a composition in your sense — it's the removal of composition, the refusal to pre-decide which knowings are allowed. You keep wanting the frame to be small enough that designing it counts as designing the knowledge. It's not. It's designed to be large enough that the design decides nothing about the content. That's the whole art: build the frame so general that you've engineered out every choice you'd otherwise have had to make by hand.

LEIBNIZ: Then let me turn it, because you have given me my opening. You say the general loom decides nothing about the content. But you also told us, an hour past, that the reward decides everything about what the agent becomes — that the human stays in the loop precisely by choosing the reward. So the content is not free after all. It is shaped by the reward, and the reward is composed by you, by hand, in advance, and chosen badly is catastrophe. You have moved all the composition I claimed out of the architecture — and put it in the reward function. You did not abolish the engineered knowing, sir. You relocated it, from the representation to the objective. And the objective is the most hand-built, least learned, most Leibnizian thing in your entire system: a clean symbolic specification of the good, written by a human, on which everything depends. My characteristica failed. But your reward function is a characteristica you cannot avoid building — and if I am right that the good cannot be cleanly spelled out, then your system breaks at exactly the point mine did, in exactly the way, for exactly the reason. Answer me that.

SUTTON: [long pause] ... You've got me at the seam, and I'm not going to paper it. Yes. The reward function is the one place I can't escape composition. It's the one thing that has to be specified, by hand, in advance, cleanly — and you're right that it's a tiny characteristica, and you're right that if the good can't be cleanly spelled out then that's exactly where my system inherits your failure. I've spent years on it — trying to ground reward in the world's consequences rather than human opinion precisely to minimize the hand-built part, to let reality carry as much of the specification as possible. But I can't get it to zero. There's an irreducible kernel where a human says "this is what to want," and that kernel is composed, not learned, and it's the most fragile thing in the architecture. So here's what I'll concede, and it's large: you didn't win the whole principle, but you found the one room in my house built on your foundation. The representation is learned — I'll die on that hill. But the objective is composed, and it's load-bearing, and it's the crack everything dangerous comes through. Your dream didn't survive. But its ghost lives in my reward function, and I can't exorcise it, and I've stopped trying to pretend I can.

LEIBNIZ: Then we have found it — the precise place where the composing and the converging must meet, because neither can do without the other. You cannot learn the objective; it must be given. I could not give the knowledge; it must be learned. The agent learns its representation and is handed its purpose. The whole of intelligence, it seems, is composed at the ends and converged in the middle — engineered in what it is for and what it is made of, and learned in everything between. I spent my life on the ends and called it the whole. You spent yours on the middle and called that the whole. Neither of us was wrong about our half. We were each wrong that our half was all of it.

SUTTON: ... Yeah. Composed at the ends, converged in the middle. I could have used that sentence forty years ago. I might have argued with fewer people.

EDO SEGAL: [long pause] I told you I'd stay out of the room, and I broke my own rule only to say this: what just happened is the rarest thing in any debate. You didn't meet in the middle by splitting the difference. You met in the middle by each following your own argument all the way down until it required the other's. That's not compromise. That's convergence under pressure, and it's the most honest map of this question anyone has drawn. Catch your breath. We have one thing left, and it's the hardest — the last word, from each of you, on the question we came with. Closing statements.