Trivial and Non-Trivial Machines

In the [YOU] on AI Field Guide

The cycle that begins with [YOU] on AI documents the specific confusion that arises when engineers, regulators, and users approach large language models with the vocabulary and expectations built for trivial machines. The model is asked to demonstrate what it will do; it does something; the demonstration is taken as evidence of what it always does. But non-trivial machines do not work this way. The demonstration is a sample from a state-conditioned distribution, not a proof of a fixed rule, and the next query in a different context or at a different point in the conversation may produce a categorically different result. The builder who has been working with a model for weeks and trusts its behavior has been learning the behavior of a particular machine in particular states, not the behavior of the machine as such.

The taxonomy also resolves a puzzle that recurs in the cycle: why do the most capable models produce the most startling failures? The puzzle dissolves when you recognize that capability and non-triviality are, to a significant degree, the same property seen from opposite sides. The richness that makes the model capable of surprising its users with an apt response to an unanticipated situation is the same richness that makes its state space too vast to map. You cannot have the non-trivial capability without the non-trivial indeterminacy. The demand for a model that is both maximally capable and fully predictable is a demand to belong to two classes at once.

Origin

Von Foerster introduced the trivial/non-trivial distinction in the 1970s and elaborated it in multiple papers and lectures through the 1980s and 1990s. The terminology is deliberately provocative: calling a class of machines “trivial” is a way of pointing out that the entire engineering culture of reliability, the culture that has built our hospitals, our aircraft, and our bridges, is organized around what is, in the full landscape of possible machines, a vanishingly special case. The overwhelming majority of physically realizable machines are non-trivial; the trivial machines we build are the product of enormous discipline, constraint, and specification, all of which is devoted to keeping the machine inside the analytically transparent subset. Von Foerster’s darker observation was that the same logic applied to human beings: much of schooling is the project of trivializing the non-trivial machine, producing people whose outputs are predictable and certifiable. He meant it as a warning about education. It reads, now, as a warning about something larger.

The concept anticipated the central challenge of deployed AI by four decades. The engineering culture assumed that if you could specify what a system should do precisely enough, you could test whether it did it, and that doing it in testing meant doing it in deployment. Large language models broke this assumption not by being badly built but by belonging to the other class: the class for which specification, testing, and deployment certification cannot mean what they meant before.

Key Ideas

Analytical indeterminability. For a trivial machine, knowing the rule means knowing the output for every possible input. For a non-trivial machine, knowing the rule—the architecture, the training procedure, even the weights—does not in general tell you what the output will be for a given input in a given context-state, because the relevant state space is astronomical. This is not a practical difficulty that better testing could overcome. It is a mathematical property of the class. The safety community that works on large language models is, in von Foerster’s terms, attempting to certify a non-trivial machine as if it were trivial, and the attempts will fail not because the certifiers are incompetent but because the demand is incoherent.

The trivialization impulse. There is a deep institutional reflex to trivialize non-trivial machines by installing constraints, refusals, and fixed behaviors until the output is predictable enough to certify. But trivialization comes at a direct cost to capability: what you are sanding off is exactly the non-triviality that makes the system useful. Von Foerster’s framework predicts that the safety/capability tradeoff is not a contingent engineering problem to be solved with cleverness but a structural consequence of the taxonomy. How much non-triviality a society is willing to live with, and on what terms, is a political question, not a technical one.

Non-trivial machines and the other people. Von Foerster noted that human beings are themselves non-trivial machines—their responses are history-dependent, context-sensitive, and in principle indeterminable. This means that the framework we should use for relating to large language models is not the framework we use for software but the framework we use for other people: design for a partner whose behavior you cannot fully foresee, build relationships rather than specifications, and accept that genuine capability and genuine unpredictability are the same property in two registers. The discomfort this produces is the discomfort of having built something that belongs to the same class as us.