CONCEPT
Loss Landscape
The high-dimensional surface defined by a neural network's training objective — the computational analog of biological fitness landscapes, whose topology determines which configurations are accessible through gradient descent.
The loss landscape is the geometric object that shapes the training of deep neural networks. Every set of parameter values defines a point in parameter space, and the loss function assigns a real number to each point representing how poorly the model performs on training data. The collection of all these values forms a surface in a space with as many dimensions as the network has parameters — typically billions. Research over the past decade has revealed that these landscapes exhibit architectural features strikingly similar to Wagner's genotype networks: extensive regions of functional equivalence, connected paths of low loss through parameter space, and diverse adjacency to alternative capabilities at every position.
In The You On AI Field Guide
The conventional narrative of neural network training describes gradient descent as hill-climbing in reverse: start at a random position and follow the gradient downhill toward minima of the loss function. The image is not wrong but incomplete. Research on mode connectivity has demonstrated that