Regularization constrains optimization so that fitting the training data is not the only objective. A model with excessive expressive power can memorize training instances, including noisy anomalies, yielding low training loss but poor test or inference performance. Regularization favors simpler solutions that retain useful fit while resisting this overfitting. section 9.3 section 9.4
For a loss L(θ), regularized optimization adds a simplicity penalty: L(θ) + λR(θ). R(θ) measures departure from simplicity, often through the length of the parameter vector, so minimizing it favors smaller absolute parameter values. The hyperparameter λ sets the trade-off: too little leaves overfitting largely unchecked; too much prevents adequate minimization of the original loss. section 9.3.1 section 9.4
| 𝕃(θ) + λR(θ) |
The objective makes the trade-off explicit: loss rewards data fit, while λR(θ) rewards a simpler parameter setting. section 9.3.1
This objective has a Bayesian interpretation. Maximum likelihood estimation (MLE) optimizes the likelihood p(X|θ), whereas maximum a posteriori (MAP) estimation optimizes p(X|θ)p(θ). The prior p(θ) supplies the regularizing preference, such as smaller parameter magnitudes. With little data, the prior has greater influence; as data accumulates, the likelihood increasingly dominates. section 6.6.3 section 6.10
Gaussian parameters that start badly
Gradient-based MLE of Gaussian μ and σ can produce very large parameters when initialized far from expected values. Adding squared μ and σ terms to the negative log-likelihood creates a MAP-style loss; the added penalty prevents parameters from becoming too large and allows convergence in the illustrated case. section 6.8.2 section 6.8.2
Regularization appears in several forms:
- Parameter penalties add a term such as a norm-based measure of parameter length to the loss, implementing the preference for compact solutions. section 9.3.1
- Dropout randomly turns off perceptrons during training, discouraging reliance on a small set of nodes and mitigating co-adaptation; it is turned off during inference. section 9.3.6
- Latent-space regularization combines reconstruction loss with a divergence from a chosen prior, encouraging compact, smooth representations rather than widely scattered encodings. section 14.6 section 14.7.6
The geometric intuition is Occam’s razor: among parameter settings that explain the data adequately, prefer the simpler one. For latent spaces, the same preference discourages unnecessarily twisted or expansive manifolds. Regularization therefore does not merely lower weights; it injects a chosen structural preference into learning. section 9.3.1 section 14.6
