Opacity of AI Systems

In the AI Story

The opacity of large language models has multiple sources. The models themselves are large neural networks with billions of parameters, whose internal computations are not accessible through direct inspection. The training procedures — pretraining on large corpora, reinforcement learning from human feedback — produce emergent behaviors that were not explicitly programmed and are not always predictable from the training setup. The interfaces through which users interact with the models further abstract away the model's operations, presenting the response without the derivation.

The research program of mechanistic interpretability is the attempt to reduce this opacity by reverse-engineering what is actually happening inside neural networks. The program has made significant progress but has not eliminated the opacity faced by users in ordinary interaction — mechanistic interpretability remains a specialized research activity that does not propagate to the user experience.

The Skinner volume treats opacity as the structural condition that makes AI-user interactions inescapably conducive to superstitious conditioning. The conditions for superstitious conditioning — reinforcement insensitive to specific response features, combined with natural behavioral variability — are permanently present. The question is not whether superstitious behaviors will develop. They will. The question is whether the development will be recognized, measured, and managed through systematic methods, or allowed to accumulate into an elaborate culture of prompting ritual.

Origin

The diagnosis of AI opacity as the structural condition for superstitious behavior development is a 2026 contribution of the Skinner volume, synthesizing the technical fact of neural network opacity with Skinner's 1948 analysis of superstitious conditioning.

Key Ideas

Opacity is structural, not incidental. The neural network architecture produces opacity that cannot be eliminated without abandoning the architecture.

Users rely on temporal contiguity to infer causation. In the absence of visible mechanism, the only inference available is from coincidence to cause.

Superstitious conditioning is therefore structural. The conditions for its development are permanent features of the interaction.

The remedy is methodological. Controlled variation can distinguish genuine effects from superstitious rituals, but requires explicit application of the method.