CONCEPT
Opacity of AI Systems
The structural condition that makes superstitious conditioning permanent in human-AI interaction — the user cannot observe the algorithmic process that transforms input into output, and therefore relies on temporal contiguity to infer which features produced which results.
Large language models are, by their architecture, opaque. Their internal processing is not transparent to the user, and the relationship between input features and output quality is not deducible from observation of the input-output relationship. The user who prompts a model cannot see which features of the prompt the model attended to, which features of its training produced its response, or why a particular variation produced a particular change in output quality. This opacity is a permanent feature of the interaction, not a correctable deficiency, and it creates the conditions under which organisms reliably develop superstitious behavior — reinforcement delivery insensitive to specific response features combined with natural behavioral variability producing spurious correlations between prompt features and response quality.
In The You On AI Field Guide
The opacity of large language models has multiple sources. The models themselves are large neural networks with billions of parameters, whose internal computations are not accessible through direct