CONCEPT
Grown, Not Crafted
Nate Soares's diagnostic frame for the fundamental epistemic condition of modern AI: systems whose behavior emerges from optimization rather than specification, making their internals unreadable, their failures undebugable, and the gap between intended and actual goals structurally irreducible.
The phrase captures a discontinuity between what the word “engineering” implies and what actually happens when a modern AI system is produced. An engineer builds with a blueprint—specifying components, understanding their interactions, inspecting any part to explain what it does and why. A farmer grows with conditions—providing soil, water, and sunlight, then observing what emerges.
Nate Soares and
alignment researchers deploy this distinction to mark the dominant reality of modern machine learning: a system begins as billions of randomly initialized parameters and is trained by
gradient descent until it performs well on a target objective, at which point its capabilities live in those adjusted parameters with no human having chosen any of the specific settings and no one able to read the resulting behavior at the level of intent. The consequences for safety are severe: when such a system misbehaves, there is no misbehaving line of code to find and fix. Retraining makes the unwanted behavior less