Regularization
Regularization is a modeling technique that constrains or penalizes solutions during fitting or optimization to discourage undesirable complexity or behavior and improve generalization or estimation.
Ensemble Methods for Machine Learning
A regularization function is a function that assigns a penalty to model parameters or complexity as part of a regularized objective.
- How it is used
- Balances data fit against model complexity to improve robustness and generalization during training.
- How deep it goes
- Defines regularized objectives and parameters, then applies L1, L2, structural, SVM, and boosted-tree penalties without formal proofs.
- Examples
- Uses RBF SVR on Friedman-1 data and noisy synthetic linear regression comparing OLS, Ridge, LASSO, and Elastic Net.
- What you can do after
- Apply it via scikit-learn models and cross-validation to select a fit-complexity tradeoff.
Read how Ensemble Methods for Machine Learning explains regularization →
Grokking Machine Learning
Regularization is a model-training technique that penalizes complexity while optimizing predictive performance.
- How it is used
- Uses a training-time complexity penalty to reduce overfitting while balancing predictive error against model complexity.
- How deep it goes
- Develops a roof-repair analogy, a penalized error expression, coefficient-shrinkage differences, and a Turi Create experiment.
- Examples
- Compares three roof repairs and three polynomial fits using no penalty, L1 regularization, and L2 regularization.
- What you can do after
- Apply it via Turi Create’s linear_regression.create with l1_penalty or l2_penalty, then inspect test predictions and coefficients.
Read how Grokking Machine Learning explains regularization →
Deep Learning with R, Third Edition
Regularization is a modeling technique that constrains learning to reduce overfitting and improve generalization to unseen data.
- How it is used
- Uses regularization after a model can overfit to improve generalization measured on validation data.
- How deep it goes
- Works through capacity reduction, weight penalties, and dropout with validation comparisons; offers no formula for optimal network size.
- Examples
- Compares IMDb review classifiers with smaller or larger hidden layers, L2 weight penalties, and dropout.
- What you can do after
- Apply it via R/Keras by changing layer size, kernel_regularizer, or layer_dropout and comparing validation loss.
Read how Deep Learning with R, Third Edition explains regularization →
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
Regularization is a technique that limits model complexity by penalizing parameter values to reduce overfitting and improve generalization.
- How it is used
- Controls overfitting in tabular models, particularly feature-rich linear regression and repeated-batch training.
- How deep it goes
- Explains coefficient penalties and contrasts L1 with L2 through evaluated scikit-learn pipelines, without deriving penalty formulas.
- Examples
- Uses Airbnb price regression with polynomial features, Ridge and Lasso pipelines, plus out-of-core logistic regression and a dropout-layer diagram.
- What you can do after
- Apply it via scikit-learn Ridge, Lasso, or SGDClassifier pipelines to reduce overfitting and select features.
Read how Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI explains regularization →
Math and Architectures of Deep Learning
The practice of constraining a learned representation or model to encourage desirable structure such as smoothness or compactness.
- How it is used
- Mitigates overfitting by adding preferences for simpler solutions alongside data-fitting loss.
- How deep it goes
- Derives penalty-based and MAP interpretations, then extends them to dropout and KL-regularized latent spaces.
- Examples
- Uses wriggly classifier boundaries, Gaussian μ and σ optimization, dropout layers, and regularized latent manifolds.
- What you can do after
- Apply it via PyTorch loss terms or dropout layers to constrain learned parameters and reduce overfitting.
Read how Math and Architectures of Deep Learning explains regularization →
Deep Learning with Python, Third Edition
A method that constrains a model during training to reduce overfitting and improve generalization.
- How it is used
- Constrains an overfitting model to improve validation performance during model development.
- How deep it goes
- Works through capacity comparisons and Keras L2 and dropout implementations; gives practical validation guidance rather than a mathematical derivation.
- Examples
- Compares four-, 16-, and 512-unit IMDb classifiers and adds L2 penalties and dropout to Keras models.
- What you can do after
- Apply it via Keras capacity changes, L2 regularizers, or dropout, selecting changes by validation performance.
Read how Deep Learning with Python, Third Edition explains regularization →
Reinforcement Learning from Human Feedback
Regularization is a technique that constrains optimization to reduce undesirable or excessive fitting to an objective.
- How it is used
- Constrains RLHF reward optimization so fine-tuned policies retain useful behavior from their reference model.
- How deep it goes
- Derives KL-penalized objectives and compares KL, PPO clipping, pretraining gradients, NLL, and margin-loss variants.
- Examples
- Uses degenerate generations, KL-versus-performance curves, clip fraction, and DPO with NLL on winning completions.
- What you can do after
- Operate/configure RLHF constraints by tuning KL strength and distinguishing reference penalties from PPO step-size clipping.
Read how Reinforcement Learning from Human Feedback explains regularization →
Deep Learning with PyTorch, Second Edition
A set of techniques that reduce overfitting and improve generalization by constraining model behavior during training.
- How it is used
- Helps a trained model generalize beyond its training set instead of overfitting.
- How deep it goes
- Defines the technique, derives the weight-decay intuition, and implements an L2 loss penalty and convolutional-network variants.
- Examples
- Uses an L2-regularized PyTorch training loop and convolutional networks with dropout or batch normalization.
- What you can do after
- Apply it via PyTorch by adding an L2 loss penalty or inserting dropout and batch normalization into a convolutional network.
Read how Deep Learning with PyTorch, Second Edition explains regularization →
Sutskever's List
A regularizer is a modeling mechanism that discourages undesirable solutions to improve generalization or control model complexity.
- How it is used
- Frames regularization as a way to curb overfitting and steer learning toward generalizable solutions, while revisiting its role in modern large models.
- How deep it goes
- Explains classical capacity control, dropout’s training mechanism, and modern overparameterized generalization, but gives no formal derivation or tuning procedure.
- Examples
- Uses polynomial fitting, AlexNet, and modular-arithmetic grokking to contrast memorization with generalization.
- What you can do after
- Evaluate how explicit regularizers and optimizer biases relate to generalization, without assuming larger capacity necessarily worsens test error.
Machine Learning Algorithms in Depth
A modeling technique that adds a penalty or constraint to discourage undesirable parameter values, such as excessive magnitude.
- How it is used
- Uses parameter penalties and constraints to select models and enlarge classification margins.
- How deep it goes
- Derives L2-norm margin optimization and regularized gradients, then implements both SVM and logistic-regression training.
- Examples
- Uses maximum-margin SVM geometry, an SVM implementation, and SGD logistic regression on a synthetic Gaussian-mixture dataset.
- What you can do after
- Build regularized SVM and logistic-regression training procedures with explicit objectives, gradients, constraints, and parameters.
Read how Machine Learning Algorithms in Depth explains regularization →
Bayesian Optimization in Action
A loss function quantifies the discrepancy between a model's predictions and desired outcomes or data, with lower values indicating better fit.
- How it is used
- Frames loss minimization as the training objective for Gaussian processes.
- How deep it goes
- Explains loss optimization, numerical instability, and variational proxies, but does not define regularization itself.
- Examples
- Uses a GPyTorch training loop, a negative marginal log likelihood, and loss curves showing numerical instability.
- What you can do after
- Operate/configure a GPyTorch GP training loop that minimizes marginal log likelihood with Adam.
Read how Bayesian Optimization in Action explains regularization →
Causal AI
Regularization is a modeling technique that adds a penalty or constraint to an optimization objective to discourage undesirable parameter values.
- How it is used
- Steers optimization toward parameter values considered better by adding penalties to a loss function.
- How deep it goes
- Defines the practice and explains L2’s squared-parameter mechanism, without developing broader regularization methods.
- Examples
- Uses L2 regularization and the sum of squared parameter values as the concrete implementation example.
- What you can do after
- Build it from scratch by appending a parameter-magnitude penalty to a chosen loss function.
Machine Learning System Design
Regularization is a technique that constrains model fitting to discourage overly complex solutions and improve generalization.
- How it is used
- Guides model-capacity decisions when training improves but validation performance deteriorates.
- How deep it goes
- Explains the overfitting pattern and recommends capacity reduction or stronger regularization, without detailing specific methods.
- Examples
- Uses diverging training and validation loss curves, plus normalization errors as a diagnostic contrast.
- What you can do after
- Diagnose overfitting from loss curves and choose between reducing complexity and increasing regularization.
Read how Machine Learning System Design explains regularization →
Related concepts
- overfittingaddresses · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.
- dropoutincludes · Dropout is a neural-network regularization technique that randomly suppresses units, connections, or activations during training to reduce overfitting.
- regularization parameterconfigured by · A regularization parameter is a model-fitting setting that controls the influence of a penalty or constraint relative to fit to the data.
- variational autoencoderused by · A variational autoencoder is a generative neural network that learns a probabilistic latent representation of data and decodes samples from it to reconstruct or generate data.
- Kullback–Leibler divergenceuses · Kullback–Leibler divergence is a nonnegative, directed measure of the information lost when one probability distribution approximates another, computed as an expected log ratio under the first distribution.
- marginal likelihoodequivalent to · Marginal likelihood, or Bayesian evidence, is the probability or density of observed data under a model after marginalizing over unknown parameters or latent variables according to their prior distributions.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.