liveBook · Manning

Weight decay

Also called L2 regularization, l2 penalty

A regularization technique that reduces model parameter magnitudes during optimization, either by adding a penalty to the objective or by directly shrinking parameters.

How 11 Manning books explain it

Grokking Machine Learning

Ridge regression is a linear regression technique that uses an L2 coefficient penalty, shrinking coefficients toward zero without usually making them zero.

How it is used
Reduces overfitting by adding a coefficient-complexity penalty to the regression error during training.
How deep it goes
Defines the L2-penalized error and illustrates gradual coefficient shrinkage with a polynomial regression comparison.
Examples
Compares unregularized, L1-regularized, and L2-regularized polynomial models on training and testing data.
What you can do after
Apply via Turi Create by setting `l2_penalty` and comparing model performance on training and testing data.

Read how Grokking Machine Learning explains weight decay →

Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI

L2 regularization adds a penalty proportional to the squared values of model parameters to constrain model complexity.

How it is used
Reduces overfitting and model complexity in regression, logistic regression, and gradient-boosting optimization.
How deep it goes
Defines squared-coefficient penalization, contrasts L1 behavior, and shows Ridge, logistic-regression, and XGBoost configurations without deriving the objective.
Examples
Uses a Ridge pipeline with polynomial features and an Optuna-tuned XGBoost regressor for Tokyo Airbnb MAE minimization.
What you can do after
Apply it via scikit-learn penalties and XGBoost lambda settings, evaluating candidate strengths with cross-validation.

Read how Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI explains weight decay →

Deep Learning with PyTorch, Second Edition

An optimization method that shrinks parameters in proportion to their current values, generally distinct from L2 regularization for adaptive optimizers.

How it is used
Uses weight decay as a regularization technique to limit parameter growth and help prevent overfitting.
How deep it goes
Explains the L2 penalty and its equivalence to proportional shrinkage under SGD, then introduces AdamW without deriving its update.
Examples
Demonstrates a PyTorch training loop with an explicit L2 penalty and constructs an AdamW optimizer for a segmentation model.
What you can do after
Apply it using PyTorch by adding an L2 penalty to the loss or selecting AdamW.

Read how Deep Learning with PyTorch, Second Edition explains weight decay →

Ensemble Methods for Machine Learning

L2 regularization penalizes the squared magnitude of model parameters or other model complexity measures in an optimization objective.

How it is used
Controls model complexity and overfitting by trading training fit against smaller parameters or simpler learners.
How deep it goes
Explains the squared-weight penalty, shrinkage, α tradeoffs, and library comparisons, but stops short of deriving optimization updates.
Examples
Uses correlated synthetic features, coefficient comparisons, test MSE and MAD, and regularized logistic-loss code.
What you can do after
Apply it via scikit-learn Ridge, compare L1 and elastic-net alternatives, and evaluate shrinkage using held-out regression metrics.

Read how Ensemble Methods for Machine Learning explains weight decay →

Deep Learning with R, Third Edition

L2 regularization is a regularization method that adds a penalty proportional to the squared values of model parameters.

How it is used
Reduces overfitting by constraining weight magnitudes and improving validation generalization.
How deep it goes
Defines the squared-weight penalty and demonstrates Keras configuration, but does not derive the optimization internals.
Examples
Uses an IMDb review classifier, Keras dense layers, validation-loss histories, and a regularizer_l2(0.002) setting.
What you can do after
Apply it via Keras by attaching kernel regularizers and comparing validation behavior during model tuning.

Read how Deep Learning with R, Third Edition explains weight decay →

Deep Learning with Python, Third Edition

How it is used
Use weight regularization to limit overfitting by discouraging large neural-network weights.
How deep it goes
Defines L2 cost as proportional to squared weights, equates it with weight decay, and demonstrates a Keras layer setting.
Examples
Uses IMDb review classification with Dense layers and an L2 coefficient of 0.002.
What you can do after
Apply L2 regularization through Keras kernel_regularizer settings while tuning for better generalization.

Read how Deep Learning with Python, Third Edition explains weight decay →

Build AI Drug Discovery Pipelines

L2 regularization penalizes the sum of squared model weights, shrinking them without generally setting them to zero.

How it is used
Controls coefficient magnitude to reduce overfitting in molecular machine-learning models with correlated features.
How deep it goes
Defines the squared-weight penalty and λ, then interprets validation curves but defers full diagnostics and hyperparameter-search details.
Examples
Uses molecular fingerprints, SVR validation curves, and VAE-CYC training with a 0.0001 learning rate.
What you can do after
Apply it via an SVR estimator by comparing validation performance across regularization strengths.

Read how Build AI Drug Discovery Pipelines explains weight decay →

Machine Learning Algorithms in Depth

A regularization method that adds a penalty proportional to the squared magnitude of model parameters to discourage large weights.

How it is used
Uses a parameter penalty to reduce overfitting in multilayer perceptrons.
How deep it goes
Defines the penalty’s purpose and parameter scope but stops short of deriving its update rule or implementation details.
Examples
Applies the penalty to nonbias weights W rather than biases b in each multilayer perceptron layer.
What you can do after
Apply it to nonbias weights when training a multilayer perceptron to discourage overfitting.

Read how Machine Learning Algorithms in Depth explains weight decay →

Causal AI

L2 regularization is a regularization method that penalizes the sum of squared parameter values.

How it is used
Steers parameter optimization toward smaller estimates by adding a squared-value penalty to the loss.
How deep it goes
Defines the penalty and its effect on large estimates, but does not derive an update rule or implementation procedure.
Examples
Uses neural-network estimation and negative log-likelihood as examples of optimization settings where regularization applies.
What you can do after
Apply L2 regularization by augmenting a loss with a sum-of-squares parameter penalty.

Read how Causal AI explains weight decay →

Timeless Algorithms, The Foundational Ideas

Weight decay is a regularization technique that penalizes large weights during model training.

How it is used
Constrains how a neural network uses its capacity during training to support generalization.
How deep it goes
Defines the loss penalty and explains its effect on reliance on large weights and generalization.
What you can do after
Understand how penalizing large weights trades some training fit for expected generalization to new observations.

Read how Timeless Algorithms, The Foundational Ideas explains weight decay →

Time Series Forecasting Using Foundation Models

L2 regularization is a penalty method that discourages large model weights by adding their squared magnitudes to the optimization objective.

How it is used
Uses ridge regression to incorporate exogenous features into TimesFM forecasts.
How deep it goes
Defines the L2 penalty and shows a ridge parameter, but stops short of deriving its optimization behavior.
Examples
Demonstrates the setting through a TimesFM covariate forecast configured with ridge=0.0.
What you can do after
Operate TimesFM's covariate forecast path by supplying a ridge setting alongside exogenous-feature inputs.

Read how Time Series Forecasting Using Foundation Models explains weight decay →

Also covered in

What to know first

Related concepts

Questions about weight decay

What is weight decay?

A regularization technique that reduces model parameter magnitudes during optimization, either by adding a penalty to the objective or by directly shrinking parameters.

Which Manning books explain weight decay?

Grokking Machine Learning, Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI, Deep Learning with PyTorch, Second Edition, Ensemble Methods for Machine Learning, Deep Learning with R, Third Edition, Deep Learning with Python, Third Edition, Build AI Drug Discovery Pipelines, Machine Learning Algorithms in Depth and 3 more.

What should I know before learning weight decay?

Manning books that teach weight decay usually introduce overfitting, objective function, regularization and linear regression first.

What is weight decay a kind of?

Weight decay is a kind of regularization. Regularization is a modeling technique that constrains learning or penalizes model complexity to discourage overfitting and improve generalization.

What is weight decay also called?

Weight decay is also called L2 regularization or l2 penalty.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.