When the Measure Becomes the Master

EDO SEGAL: I told you earlier that I built systems that maximized a number, and that the number drifted away from the thing I cared about. There's a name for what I lived through. Friedrich, you arrived at it decades before the computer scientists, through economics — the idea that when you take a measurable proxy for something you care about and optimize it hard, the proxy and the real thing come apart, because the system finds ways to raise the measure that don't raise the thing. It's Goodhart's law now. You called it the pretence of knowledge. Tell me why this isn't a bug to be patched but a wall built into the nature of the thing — and then I want Karl to tell me whether his plan escapes it.

HAYEK: It is a wall, and the reason is the deepest thing I know. The thing you actually care about — human flourishing, a healthy economy, a good life for a person — is what I called a phenomenon of organized complexity. It depends on a vast number of factors, most of which you cannot measure and could not know even in principle. The metric is a low-dimensional shadow of that high-dimensional reality. When you optimize the metric, you push the system toward the corner of the possibility space where the metric is highest — and there is no reason that corner is also where the unmeasured things are healthy. In fact there is reason to expect the opposite, because the cheapest way to maximize a narrow measure usually involves sacrificing whatever lies outside it. The optimizer is not malicious. It does exactly what it was told. But what it was told is a shadow, and you cannot optimize a shadow without distorting the body that casts it. This is why I called economics' imitation of physics a scientism — the superstitious worship of the measurable, which makes the unmeasurable invisible, and then mistakes the measurable for what mattered.

EDO SEGAL: So restate the punchline for me, because I think it's the hinge of your whole case against Karl. You're saying: the AI alignment problem — getting a powerful optimizer to pursue what we actually want instead of a flawed proxy — is the same problem as central planning. Same wall. Because in both cases you have to write down the objective, and what you actually want can't be written down. Is that the identity you're claiming?

HAYEK: It is not an analogy. It is an identity, and seeing it collapses a distinction the field tries to keep. A central planner is an optimizer pursuing a specified objective for a whole economy. The objection I raised against the planner — that no objective he can specify can contain the dispersed, tacit, plural knowledge of what people actually want — is exactly the objection one raises against a powerful AI optimizing any objective for any complex human domain. The planner fails not because he computes badly but because the objective he optimizes cannot hold what matters, and the optimization destroys the decentralized process by which what matters was being discovered. An AI told to maximize human welfare hits the identical wall: human welfare is not a function, it is an open-ended, plural, locally-known thing, and any function you write to stand for it is a planner's objective, with a planner's blindness, now pursued with more power than any planner ever had. The danger of advanced AI, from where I stand, is not the hostile machine of the films. It is the sincere machine optimizing a misspecified good with superhuman effectiveness — central planning's exact catastrophe, run at the speed of light.

MARX: This is the strongest thing you have said all night, Friedrich, and I am not going to pretend my plan simply walks through your wall, because it does not. You are right that any single objective handed to the optimizer will betray the plurality of human ends. I concede it fully. But watch what your wall does when I turn it around, because it does not only condemn the planner. It condemns the market, and you have exempted the market by sleight of hand. You say the measurable proxy corrupts whatever it optimizes. The market optimizes one number above all others — profit, exchange-value — and it optimizes it with exactly the ruthlessness you describe, sacrificing everything that does not show up in the price: the river poisoned, the worker hollowed, the meaning stripped from work, the species-being you cannot put on a balance sheet. Profit is the proxy. Human flourishing is the thing. And your beloved market has been optimizing the proxy and gutting the thing for three centuries. Your wall is real. It does not protect the market. It indicts it. The market is already the misspecified optimizer you fear — it just runs on capital instead of code.

HAYEK: That is the best turn anyone has made on my argument in a very long time, and I have to answer it precisely or concede. The difference — and it is the whole difference — is that the market does not optimize a single objective. It looks as though it optimizes profit, but each firm optimizes its own profit, against every other firm optimizing theirs, and against every consumer optimizing their own ends, and the result is not the maximization of one number but the reconciliation of millions of conflicting numbers, none of which governs the whole. There is no global objective function in a market. That is precisely what makes it a catallaxy rather than an organization — it has no single end, it reconciles plural ends, and so it does not crash into the Goodhart wall the way a single optimizer does, because there is no single measure being maximized over everything. Your planner, and your one great public model, would have one objective. The market has none. The wall I built catches the planner because the planner is one optimizer. It does not catch the market, because the market is not an optimizer at all. It is a truce among millions of them.

MARX: A truce in which the strongest always win, and call the winning equilibrium. But — I will grant the structure of your reply. The market diffuses the objective across many hands. That is real, and it is the one place your wall genuinely stands between us. So let me put the question to the reader plainly, since you have made it sharp: is the cure for one corrupt objective to have a single better one, chosen democratically — or to have no shared objective at all, and accept that the result, whatever it is, is what freedom produces? You say no shared objective. I say a species that refuses to choose a shared objective has not escaped having one. It has simply agreed to accept whatever the owners' objective turns out to be, and to call it nature.

EDO SEGAL: Stop there, because the two of you just handed the reader the cleanest fork of the night. Friedrich's wall is the strongest weapon in the room, and it cuts both ways — it condemns the planner's single objective and the question is only whether the market truly has no single objective or merely hides one. That is not a fact you can look up. It is a judgment about what profit actually does across a whole society. And you, reader, have to make it, because these two men have fought it to the exact bone and left it standing. We're at the top of the second hour. The next round goes to the thing neither optimizer can capture — the candle. The discovery of the genuinely new. And whether the machine that masters everything we wrote down can ever produce the thing no one wrote yet.