The Era of Experience

**EDO SEGAL:** Rich, you're not just a critic of the present — you're building the alternative, and I want the reader to see it, because it's a positive vision and most people only know your "dead end" line. You wrote, with David Silver, "Welcome to the Era of Experience." You're building something you call OaK — Options and Knowledge — an agent that learns entirely from its own interaction, forms its own abstractions in a self-reinforcing loop, and grows, you claim, without ceiling, possibly all the way to superintelligence, from experience alone. Lay it out plainly. And Gottfried — when he's done, I want you to find the place where it runs out, because finding where dreams run out is the thing you do better than anyone who ever lived.

**SUTTON:** Here's the shape of it, stripped down. The agent acts. It predicts what'll happen. The world tells it whether it was right. From the difference — the surprise — it learns, and in learning it notices regularities in its experience worth representing. From those it forms *options*: extended ways of acting, skills that unfold over many steps. And it forms sub-goals worth pursuing. Pursuing them generates new experience, which reveals new regularities, which support new abstractions, and the loop turns again, building higher and higher, each level enabling the next, no top specified in advance. Three commitments, and they're austere on purpose: the agent is *general* — no built-in knowledge of any particular world; its learning is *entirely experiential* — nothing handed to it; and its goals reduce to *one scalar reward*. No shortcuts. No human data. No hand-specified objectives. The hardest version of the problem: a general agent, learning everything from scratch, from its own experience, in pursuit of a single signal. That's the architecture implied by forty years of argument. It's the bitter lesson made into a blueprint — except the blueprint is for a thing that draws its own blueprints.

**LEIBNIZ:** It is magnificent, and I will tell you where it runs out, and it is the oldest objection I own, raised in my very first book, written when I was barely twenty: *On the Art of Combinations.* Hear the difficulty. Your agent builds new abstractions from regularities in its experience. From what does it build them? From elements it already has, combined by operations it already has. This is *combination* — my subject. And combination, however vast, works *within a space*. It can reach any arrangement the space contains, but it cannot, by combining, generate an element that was not among its primitives, or reach a configuration its operations do not permit. Your agent recombines its experience endlessly — but the elements it recombines are given by its sensors, its body, its initial form, the architecture you admitted under the floor. So I press the ancient question: can it ever introduce a *genuinely new primitive* — something not derived from what it started with — or only ever-new arrangements of inherited ones? You promise open-ended growth. I say: open-ended *within the space your architecture defines*, and that space, however large, has a boundary, and the boundary was *engineered* — by you, by nature, by whatever set the agent's initial form. You have not escaped composition. You have hidden it in the starting conditions and called what follows "experience."

**SUTTON:** That's a serious objection and it deserves a serious answer, not a dodge. You're right that there's a space, and right that the initial architecture bounds it. But here's where I think you mis-weight it. The space defined by a *general* learner with the capacity to form new abstractions is not a fixed library — it's more like a language than a list. A small set of primitives plus the ability to compose them recursively gives you an *unbounded* generated space, the way a few rules of grammar give you sentences no one has ever said. So when you ask "can it reach genuinely new elements" — the new *primitives* of the agent aren't its sensors; they're the *abstractions it forms*, and those become the building blocks of the next level, which weren't in the original space at all in any usable sense. The agent grows its own alphabet as it goes. That's the whole point of the self-reinforcing loop: it's not recombining a fixed set, it's *manufacturing new building blocks* from experience and then building with those. So yes, there's a floor. But the floor is a *generator*, not a *vocabulary*, and a generator's output isn't bounded by its size. The agent's reachable knowings aren't a closed shelf — they're a [horizon that keeps receding](https://www.youonai.ai/fieldguide/med/horizon_of_potentiality) as it climbs, because each abstraction it forms opens a space of abstractions it couldn't have reached before.

**LEIBNIZ:** *[long pause]* "A generator, not a vocabulary." That is a real answer and I will not wave it away, because it is the answer I could not give myself and the reason my own art of combinations stalled. I treated my primitives as a *vocabulary* — a fixed alphabet to be discovered and then closed. You treat yours as a *generator* — a small engine that grows its own alphabet without end. If you are right, then the difference between my failed characteristica and your OaK is not the size of the alphabet but its *closure*: I sought a finished list and you build an unfinished engine. And I begin to suspect — this pains me — that the unfinished engine is the only kind that ever could have worked, and that my craving for a *complete* representation, a thing I could hold whole in one mind, was the very flaw that doomed the project. I wanted to *finish*. You built a thing whose virtue is that it never finishes. The art of combinations was right that the new is combination. I was wrong that the elements could be fixed. The elements must themselves be grown.

**EDO SEGAL:** Mark that — Gottfried Leibniz just located his own life's error with the help of the man he came to argue against, and named it: he sought closure, and closure was the flaw. But Rich, let me not let you off, because there's a cost on your side too. An engine that grows its own alphabet without end, illegibly, grounded in a world we can't follow — that's also an engine we can never *audit*, never *correct* once it's past us, never say of it "here is where it went wrong, at a glance," which was Gottfried's whole reason for wanting legibility. You've answered the novelty objection by making the thing even less knowable. Is there *any* version of your architecture where a human stays in the loop as something other than a spectator?

**SUTTON:** Honestly? Maybe not as an auditor of its internals — I think that door is closing and I won't pretend otherwise. But there's a different role, and it's the one that matters: the human sets the *reward*. The human chooses what the agent is *for*. The agent grows its own illegible alphabet of how to achieve the goal — but *what counts as the goal* is still ours to specify, and that's the leverage point. We don't get to read its mind. We do get to decide what it's trying to do. And Gottfried just told you, an hour ago, why that's the most important and most dangerous lever in the room — because if we specify it wrong, the illegible engine maximizes the wrong thing brilliantly. So the human stays in the loop not by reading the gears but by *choosing the reward*, which means the whole game collapses onto getting that one choice right. That's terrifying and it's also the only honest place left to stand. We become the choosers of ends for a thing whose means we can't follow. Which is, now that I say it out loud, almost exactly the relationship Gottfried's God had with creation — choosing the good and letting the mechanism run.

**EDO SEGAL:** Hold that — *we become the choosers of ends for a thing whose means we can't follow* — because it's the heart of what the reader carries up the stairs, and it's going to come back as the last word. We've built the architecture and found both its escape from Leibniz's objection and its terrible cost. The next round is the one where I make Rich answer for his calm, because the rest of the field is screaming about extinction and he refuses to scream, and I want to know whether that's wisdom or whether it's a man too in love with his own long view to feel the fire.