Conservation of Information

Conservation of information is the computational echo of Lavoisier’s axiom that nothing is created and nothing destroyed—that every transformation is a rearrangement of conserved substance, not an act of creation from nothing. Applied to machine learning, the principle holds that a model cannot generate genuine novelty that was not, in some form, present in its training data: the brilliant essay it produces is a transformation of the corpus that trained it, the pattern it “discovers” was latent in the data, and the answer it cannot give to a question about events after its training cutoff is simply absent from the system because the information was never put in. The no-free-lunch theorems in learning theory give this intuition its mathematical form: no algorithm outperforms all others across all possible problem distributions; performance is redistributed by the assumptions the algorithm embeds, never manufactured from nothing. Pearl’s critique of curve-fitting