Regularization
Regularization is a modeling technique that constrains learning or penalizes model complexity to discourage overfitting and improve generalization.
Ensemble Methods for Machine Learning
A regularization function is a function that assigns a penalty to model parameters or complexity as part of a regularized objective.
- How it is used
- Balances data fit against model complexity to improve robustness and generalization during training.
- How deep it goes
- Defines regularized objectives and parameters, then applies L1, L2, structural, SVM, and boosted-tree penalties without formal proofs.
- Examples
- Uses RBF SVR on Friedman-1 data and noisy synthetic linear regression comparing OLS, Ridge, LASSO, and Elastic Net.
- What you can do after
- Apply it via scikit-learn models and cross-validation to select a fit-complexity tradeoff.
Read how Ensemble Methods for Machine Learning explains regularization →
Grokking Machine Learning
Regularization is a model-training technique that penalizes complexity while optimizing predictive performance.
- How it is used
- Uses a training-time complexity penalty to reduce overfitting while balancing predictive error against model complexity.
- How deep it goes
- Develops a roof-repair analogy, a penalized error expression, coefficient-shrinkage differences, and a Turi Create experiment.
- Examples
- Compares three roof repairs and three polynomial fits using no penalty, L1 regularization, and L2 regularization.
- What you can do after
- Apply it via Turi Create’s linear_regression.create with l1_penalty or l2_penalty, then inspect test predictions and coefficients.
Read how Grokking Machine Learning explains regularization →
Deep Learning with R, Third Edition
Regularization is a modeling technique that constrains learning to reduce overfitting and improve generalization to unseen data.
- How it is used
- Improves generalization once a model can overfit, using constraints guided by validation performance.
- How deep it goes
- Explains capacity reduction, weight regularization, and dropout with model comparisons and validation guidance.
- Examples
- Compares IMDb movie-review classifiers using smaller networks, L2 weight regularization, and dropout.
- What you can do after
- Apply regularization via R/Keras and compare model variants using validation loss.
Read how Deep Learning with R, Third Edition explains regularization →
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
Regularization is a technique that limits model complexity by penalizing parameter values to reduce overfitting and improve generalization.
- How it is used
- Controls overfitting in tabular models, particularly feature-rich linear regression and repeated-batch training.
- How deep it goes
- Explains coefficient penalties and contrasts L1 with L2 through evaluated scikit-learn pipelines, without deriving penalty formulas.
- Examples
- Uses Airbnb price regression with polynomial features, Ridge and Lasso pipelines, plus out-of-core logistic regression and a dropout-layer diagram.
- What you can do after
- Apply it via scikit-learn Ridge, Lasso, or SGDClassifier pipelines to reduce overfitting and select features.
Read how Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI explains regularization →
Grokking Machine Learning, Second Edition
Regularization is a modeling technique that constrains parameter values or model complexity to reduce overfitting.
- How it is used
- Reduces overfitting by balancing prediction error against model complexity during training.
- How deep it goes
- Explains modified errors, λ tuning, L1/L2 coefficient shrinking, and scikit-learn use; formal gradient-descent mathematics is deferred to an appendix.
- Examples
- Uses roof repairs, movie recommendations, coefficient shrinking, and degree-20 polynomial fitting with Lasso and Ridge.
- What you can do after
- Apply it via scikit-learn Lasso or Ridge by selecting alpha and comparing regularized polynomial models on held-out data.
Read how Grokking Machine Learning, Second Edition explains regularization →
Math and Architectures of Deep Learning
The practice of constraining a learned representation or model to encourage desirable structure such as smoothness or compactness.
- How it is used
- Mitigates overfitting by adding preferences for simpler solutions alongside data-fitting loss.
- How deep it goes
- Derives penalty-based and MAP interpretations, then extends them to dropout and KL-regularized latent spaces.
- Examples
- Uses wriggly classifier boundaries, Gaussian μ and σ optimization, dropout layers, and regularized latent manifolds.
- What you can do after
- Apply it via PyTorch loss terms or dropout layers to constrain learned parameters and reduce overfitting.
Read how Math and Architectures of Deep Learning explains regularization →
Build AI Drug Discovery Pipelines
Regularization is a modeling technique that constrains learning to reduce overreliance on particular features and improve performance on unseen data.
- How it is used
- Uses regularization to improve generalization in molecular screening, decision trees, and drug–target affinity prediction.
- How deep it goes
- Explains coefficient penalties, tree constraints, dropout, and parameter sharing; defers full coefficient diagnostics and tuning configurations.
- Examples
- Illustrates regularization with hERG fingerprint coefficients, decision-tree pruning, and dropout in drug–target affinity prediction.
- What you can do after
- Evaluate regularization approaches and identify controls for coefficient penalties, decision trees, and neural networks.
Read how Build AI Drug Discovery Pipelines explains regularization →
Deep Learning with Python, Third Edition
A method that constrains a model during training to reduce overfitting and improve generalization.
- How it is used
- Uses regularization after establishing an overfitting model to improve generalization during model development.
- How deep it goes
- Explains capacity reduction, L2 regularization, dropout, and a variational autoencoder loss through practical model examples.
- Examples
- Compares IMDb classifiers with different hidden-layer sizes and demonstrates L2 and dropout variants in Keras.
- What you can do after
- Apply regularization via Keras and compare model choices using validation performance.
Read how Deep Learning with Python, Third Edition explains regularization →
Reinforcement Learning from Human Feedback
Regularization is a technique that constrains optimization to reduce undesirable or excessive fitting to an objective.
- How it is used
- Controls reward-driven fine-tuning of a capable model to limit over-optimization and preserve useful starting behavior.
- How deep it goes
- Derives KL-penalized and auxiliary-loss objectives and explains PPO clipping, while distinguishing explicit penalties from implicit on-policy effects.
- Examples
- Uses InstructGPT pretraining gradients, Llama 2 reward margins, and PPO clip fraction to illustrate different controls.
- What you can do after
- Evaluate regularization choices and their trade-offs when fine-tuning a language model against a reward signal.
Read how Reinforcement Learning from Human Feedback explains regularization →
Deep Learning with PyTorch, Second Edition
A set of techniques that reduce overfitting and improve generalization by constraining model behavior during training.
- How it is used
- Uses regularization to help convolutional models generalize beyond their training data.
- How deep it goes
- Explains L1 and L2 penalties and the weight-decay interpretation, then demonstrates dropout and batch normalization in convolutional networks.
- Examples
- Uses an L2-penalized training loop and convolutional networks with dropout and batch normalization.
- What you can do after
- Apply it via PyTorch by adding an L2 penalty to a training loss and incorporating dropout or batch normalization layers.
Read how Deep Learning with PyTorch, Second Edition explains regularization →
Sutskever's List
A regularizer is a modeling mechanism that discourages undesirable solutions to improve generalization or control model complexity.
- How it is used
- Frames regularization as a way to curb overfitting and steer learning toward generalizable solutions, while revisiting its role in modern large models.
- How deep it goes
- Explains classical capacity control, dropout’s training mechanism, and modern overparameterized generalization, but gives no formal derivation or tuning procedure.
- Examples
- Uses polynomial fitting, AlexNet, and modular-arithmetic grokking to contrast memorization with generalization.
- What you can do after
- Evaluate how explicit regularizers and optimizer biases relate to generalization, without assuming larger capacity necessarily worsens test error.
Grokking Bayes
Regularization is a modeling technique that constrains or penalizes parameter configurations to reduce overfitting and control model complexity.
- How it is used
- Controls model complexity and limits overfitting when data are small or noisy.
- How deep it goes
- Explains weight penalties, prior-based shrinkage, L₂ equivalence, and hierarchical pooling without deriving optimization or posterior calculations.
- Examples
- Uses house-price features, zero-centered weights, and pooled school coaching effects to illustrate shrinkage and variance reduction.
- What you can do after
- Evaluate/understand how penalties, priors, and pooling produce simpler and more conservative predictive models.
Machine Learning Algorithms in Depth
A modeling technique that adds a penalty or constraint to discourage undesirable parameter values, such as excessive magnitude.
- How it is used
- Uses parameter penalties and constraints to select models and enlarge classification margins.
- How deep it goes
- Derives L2-norm margin optimization and regularized gradients, then implements both SVM and logistic-regression training.
- Examples
- Uses maximum-margin SVM geometry, an SVM implementation, and SGD logistic regression on a synthetic Gaussian-mixture dataset.
- What you can do after
- Build regularized SVM and logistic-regression training procedures with explicit objectives, gradients, constraints, and parameters.
Read how Machine Learning Algorithms in Depth explains regularization →
Bayesian Optimization in Action
A loss function quantifies the discrepancy between a model's predictions and desired outcomes or data, with lower values indicating better fit.
- How it is used
- Frames loss minimization as the training objective for Gaussian processes.
- How deep it goes
- Explains loss optimization, numerical instability, and variational proxies, but does not define regularization itself.
- Examples
- Uses a GPyTorch training loop, a negative marginal log likelihood, and loss curves showing numerical instability.
- What you can do after
- Operate/configure a GPyTorch GP training loop that minimizes marginal log likelihood with Adam.
Read how Bayesian Optimization in Action explains regularization →
Causal AI
Regularization is a modeling technique that adds a penalty or constraint to an optimization objective to discourage undesirable parameter values.
- How it is used
- Steers optimization toward parameter values considered better by adding penalties to a loss function.
- How deep it goes
- Defines the practice and explains L2’s squared-parameter mechanism, without developing broader regularization methods.
- Examples
- Uses L2 regularization and the sum of squared parameter values as the concrete implementation example.
- What you can do after
- Build it from scratch by appending a parameter-magnitude penalty to a chosen loss function.
Timeless Algorithms, The Foundational Ideas
Regularization is a method that constrains or penalizes a model to limit complexity and improve estimation, especially when data is limited.
- How it is used
- Limits overfitting by reducing a model’s dependence on training-specific details.
- How deep it goes
- Explains capacity constraints and SVM slack penalties, relating regularization strength to bias and variance without prescribing a universal setting.
- Examples
- Uses neural-network weight decay and the soft-margin SVM penalty parameter C as examples.
- What you can do after
- Evaluate regularization by balancing penalized violations against margin width and recognizing overfitting or underfitting tendencies.
Read how Timeless Algorithms, The Foundational Ideas explains regularization →
Machine Learning System Design
Regularization is a technique that constrains model fitting to discourage overly complex solutions and improve generalization.
- How it is used
- Guides model-capacity decisions when training improves but validation performance deteriorates.
- How deep it goes
- Explains the overfitting pattern and recommends capacity reduction or stronger regularization, without detailing specific methods.
- Examples
- Uses diverging training and validation loss curves, plus normalization errors as a diagnostic contrast.
- What you can do after
- Diagnose overfitting from loss curves and choose between reducing complexity and increasing regularization.
Read how Machine Learning System Design explains regularization →
Also covered in
- Privacy-Preserving Machine Learning
- Evolutionary Deep Learning: Genetic algorithms and neural networks
- Building Specialized AI SystemsIn early access (MEAP)
- Designing Deep Learning Systems: A guide for software engineers
- Transformers in Action
- Machines that ThinkIn early access (MEAP)
- AI Recommender SystemsIn early access (MEAP)
What to know first
- overfittingOverfitting is a modeling condition in which a model adapts too closely to particular data, including its noise or idiosyncrasies, and consequently generalizes poorly to new data.
- objective functionAn objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- gradient descentGradient descent is an iterative optimization method that updates parameters in the direction opposite an objective function’s gradient to reduce the objective’s value.
Related concepts
- overfittingaddresses · Overfitting is a modeling condition in which a model adapts too closely to particular data, including its noise or idiosyncrasies, and consequently generalizes poorly to new data.
- dropoutincludes · Dropout is a neural-network regularization technique that randomly suppresses selected units or activations during training to reduce overfitting.
- regularization parameterconfigured by · A regularization parameter is a model-fitting setting that controls the influence of a penalty or constraint relative to fit to the data.
- variational autoencoderused by · A variational autoencoder is a generative neural network that learns a probabilistic latent representation of data and decodes samples from it to reconstruct or generate data.
- elastic netincludes · Elastic net is a regression regularization method that combines L1 and L2 penalties on model coefficients.
- marginal likelihoodequivalent to · Marginal likelihood, or Bayesian evidence, is the probability or density of observed data under a model, averaged over its unknown parameters or latent variables according to their prior distributions.
Questions about regularization
What is regularization?
Regularization is a modeling technique that constrains learning or penalizes model complexity to discourage overfitting and improve generalization.
Which Manning books explain regularization?
Ensemble Methods for Machine Learning, Grokking Machine Learning, Deep Learning with R, Third Edition, Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI, Grokking Machine Learning, Second Edition, Math and Architectures of Deep Learning, Build AI Drug Discovery Pipelines, Deep Learning with Python, Third Edition and 9 more.
What should I know before learning regularization?
Manning books that teach regularization usually introduce overfitting, objective function, language model and gradient descent first.
What kinds of regularization are there?
Kinds of regularization include dropout and elastic net.
What is regularization also called?
Regularization is also called regularize.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
















