CONCEPT
The AI Opacity Barrier
The structural property of large language models by which the reasoning behind their outputs is not inspectable in the form a human reviewer would need to evaluate it — extending structural secrecy from the organization into the tool itself and producing a form of secrecy that no organizational reform can eliminate.
The AI opacity barrier is the technological component of structural secrecy that Vaughan's original framework did not anticipate. Large language models generate output through statistical processes operating over billions of parameters, distributed across architectures that do not decompose into the sequential, inspectable chain of decisions characterizing human reasoning about code. The developer can observe the output, test the output, evaluate whether the output does what it is supposed to do. What she cannot do is inspect why the model made specific design choices, what alternatives it considered, what assumptions it embedded, or what conditions might cause those assumptions to fail.
In The You On AI Field Guide
The barrier is not a flaw in current models that future versions will correct. It is a structural property of the technology: the reasoning, in the sense that a human reviewer would