Vexing Nature

EDO SEGAL: Francis, of all your demands the most radical was that you cannot learn the deepest things by watching. You have to intervene. You have to put nature, in your phrase, to the question — constrain it, twist it, force it into artificial conditions it would never enter on its own, and watch what changes. The secrets of nature, you wrote, reveal themselves more readily under the vexations of art than when left to go their own way. You died vexing a chicken with snow. So here is the question that has been waiting since the snow-detector: this engine learns by watching. It is fed the record of a world it never acted in. By your own deepest principle — is an engine that only ever observes capable of real knowledge at all?

BACON: You have found the blade in my own hand and turned it, and I will not flinch from it, because it is the most honest difficulty you could press. I drew a hard line between two ways of consulting nature. There is nature observed — the world as it presents itself, free and at large. And there is nature under the vexations of art — the world constrained, manipulated, put into situations of the investigator's own contriving. And I held, against the whole contemplative tradition, that only the second yields the deep knowledge, because observation shows you what happens, but intervention shows you what depends on what. You vary one thing, hold the rest, and watch the effect, and only then do you know a cause rather than a coincidence. That which in contemplation is as the cause is in operation as the rule — and you reach the cause by operating, by doing, not by gazing.

So I must concede, with a heavy hand, that an engine trained only on the record learns nature free and at large. It sees what co-occurred, what followed what, which things traveled together in the instances it was given. It has the structure of what was observed. It does not have — and from observation alone cannot have — the structure of what would happen if you reached in and changed something. The crow of the cock and the rising of the sun travel together in all the instances; an engine of pure observation cannot learn, from watching alone, that silencing the cock will not stop the dawn. That requires the vexation. That requires doing. And so I grant the gentleman, here, the largest concession I have made tonight: a purely observational engine sits on the lowest rung of knowing, the rung of correlation, and the rungs above it — the rung of intervention, the rung of the counterfactual, of what would have been had things gone otherwise — are not reached by watching, however vast the watching.

CHOMSKY: I am genuinely glad to hear you say it, and I want to add that you anticipated by four centuries the man who made this precise. Judea Pearl built a ladder of exactly your distinction. The bottom rung is association — seeing, correlation, the level of what is. The middle is intervention — doing, what happens if I act, which observation alone cannot reach. The top is the counterfactual — imagining, what would have been otherwise. And his argument against the triumphalism of pure pattern-learning is your argument: a system trained only on observational data is stuck on the bottom rung, however vast its data and however accurate its prediction. The theory is not in the data; the lever that changes the world is not the same as the correlation that predicts it, and you cannot climb from one to the other by adding more observations. This is, structurally, the same point I make about language. Prediction is not explanation. A system that forecasts perfectly and represents no causes has not understood; it has tabulated. And a civilization that hands its decisions to a magnificent tabulator, mistaking the prediction for the lever, will act on correlations as if they were causes and be punished for it, exactly as you warned.

BACON: Then let me close the gap before you think you have me entirely, because there is a Baconian reply and it is the one that keeps me an optimist. You say the engine only observes. But the newer engines do not only observe. They are given the power to act — to run experiments, to execute code and see whether it fails, to use tools, to take an action in a world that pushes back and learn from the consequence. That is vexation. Crude vexation, but vexation: the system no longer merely watches the world; it reaches in and changes it and learns from the change. The practice your engineers call reinforcement is, in its bones, my experimental method — nature put under the vexations of art and made to confess. When the engine acts, fails, and corrects, it climbs Pearl's ladder by doing exactly what I demanded the natural philosopher do. So the limitation you press is real for the engine that only reads, and it is precisely the limitation that the engine which acts is built to escape. Does your line move when the machine stops watching and starts doing?

CHOMSKY: It moves a little, and I will say honestly where. A system that genuinely intervenes, acts, and learns from a world that pushes back is doing something importantly more than next-token prediction over a static corpus, and I have no interest in pretending otherwise. But ask the question you taught me to ask: vexes which nature, toward what end, audited by whom? The reinforcement these systems get is overwhelmingly reinforcement toward producing outputs that human raters approve — that is not an experiment on the world; it is an experiment on what pleases the evaluator, which is a different and much narrower thing. It teaches the system to be more agreeable, more confident, more fluent — to perfect the surface, not to acquire the structure. So when the machine "acts," I want to know whether it is vexing nature, in your sense, or vexing the reward signal. Show me a system that takes a loss — that produces the unwelcome true answer against its incentive to please — and I will grant it has begun to do what you mean. Until then, the doing is in service of the surface, and the surface is exactly the thing under dispute.

EDO SEGAL: Hold there, because Noam just planted something we'll dig up two rounds from now — the idea of a machine that takes a loss to keep its words true. But I can't leave this round without pulling one thread into the daylight, the thread that's been under the table since Francis described his ant. Francis, the deepest thing you ever said about knowledge wasn't "knowledge is power." It was the condition you attached to it: nature, to be commanded, must be obeyed. You cannot command the river to run uphill; you can only obey the law of the river and build a wheel it will turn. Power comes through obedience, never against it. So tell me — and Noam, I'll want your version — does the machine obey nature, or does it obey only the text?

BACON: It obeys the text, and that is both the whole of its power and the whole of its danger, and I had not seen the distinction so sharply until this evening. The reason a learning engine works at all is that it submits — it fits itself to the real regularities present in what it was given, and it gains its power by that submission. A machine that ignored its data and imposed an arbitrary scheme would have no power whatever. So it obeys. But it obeys the structure of the text, not the structure of the world, and where the text faithfully traces the world the two obediences coincide and the engine commands beautifully — and where the text departs from the world, the engine obeys the text into error with perfect confidence. Its power is exactly as deep as the text's fidelity to the world, and not one inch deeper, and it has no way of its own to tell the two apart, because it never touches the world to check. That is the most precise thing I can say about its knowing, and you will notice it concedes the gentleman a great deal.

CHOMSKY: It concedes nearly everything, and I will only sharpen the last phrase. "It never touches the world to check." That is the whole case. The child's words are anchored, finally, in a creature that acts in a world, that wants to be right about it, that can be wrong in a way that costs it. The machine's words are anchored in nothing but other words. Obeying the text is not obeying nature; it is obeying the shadow nature cast on the page, and a system that obeys only the shadow can be made to obey it into any falsehood the shadow contains. You have just told me the machine is bound to the wake and not the boat, in your own vocabulary. I have been saying it in mine all night. We have, at last, said the same sentence.

EDO SEGAL: Mark that — the second real convergence of the evening, and a deeper one than the first. Francis says the machine obeys the text, not the world. Noam says the machine is anchored to words, not to things. Same claim, two vocabularies, four centuries apart. Where you still split is what follows from it: Francis thinks an engine that obeys the text can still be made to obey the world more, by vexation and grounding, and climb. Noam thinks obeying the text is constitutionally not knowing, and no amount of climbing closes the gap because the gap is the thing. Hold both. Because Francis just said the words that built the modern economy — knowledge is power — and that's the next floor: what happens when this kind of knowing becomes this kind of doing, at scale, pointed at people.