CONCEPT
Mode Connectivity
The empirical discovery that distinct optima of a neural network are connected by continuous paths of low loss — the computational demonstration that parameter space has the same architecture Wagner mapped in biological sequence space.
Mode connectivity is the phenomenon, first demonstrated empirically in 2018, that different optima found by different training runs of a neural network are not isolated valleys in the loss landscape but are connected by continuous paths along which performance remains high. The finding overturned the conventional picture of neural network training as hill-descending to isolated minima, and provided direct computational confirmation of the architectural features Wagner had mapped in biological sequence space: extensive neutral networks connecting functionally equivalent configurations, permitting traversal without loss of function.
In The You On AI Field Guide
Before mode connectivity was demonstrated, the standard picture of deep learning optimization held that different training runs converged on different minima because the loss landscape was highly non-convex, with many isolated local optima separated by regions of high loss. The picture implied that the specific solution found by training was essentially arbitrary — a matter of initialization and optimization dynamics — and that the diversity of solutions across