CONCEPT
Gradient Descent
The directed optimization algorithm that navigates neural network parameter space by following the slope of the loss function — structurally unlike biological mutation, but whose trajectories nonetheless traverse the same kind of topological architecture Wagner mapped.
Gradient descent is the primary mechanism by which neural networks are trained. It adjusts parameters along the direction of steepest descent of the loss function, iteratively reducing training error. Unlike biological mutation — which is random and undirected — gradient descent is guided by a signal: the gradient itself, which indicates the direction in parameter space that most rapidly decreases loss. This directedness is a fundamental disanalogy with Wagner's biological framework, where the randomness of mutation is essential to the argument that topology must make innovation accessible through undirected search. Yet gradient descent interacts with loss landscapes whose architecture exhibits the same features Wagner identified in sequence space — suggesting that directed search through structured topology produces dynamics both similar to and distinct from undirected biological exploration.
In The You On AI Field Guide
The classical picture of gradient descent treats it as hill-descending: from any starting position in parameter space, follow the negative gradient until reaching a minimum.