Weight decay
A regularization technique that reduces model parameter magnitudes during optimization, either by adding a penalty to the objective or by directly shrinking parameters.
Grokking Machine Learning
Ridge regression is a linear regression technique that uses an L2 coefficient penalty, shrinking coefficients toward zero without usually making them zero.
- How it is used
- Reduces overfitting by adding a coefficient-complexity penalty to the regression error during training.
- How deep it goes
- Defines the L2-penalized error and illustrates gradual coefficient shrinkage with a polynomial regression comparison.
- Examples
- Compares unregularized, L1-regularized, and L2-regularized polynomial models on training and testing data.
- What you can do after
- Apply via Turi Create by setting `l2_penalty` and comparing model performance on training and testing data.
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
L2 regularization adds a penalty proportional to the squared values of model parameters to constrain model complexity.
- How it is used
- Reduces overfitting and model complexity in regression, logistic regression, and gradient-boosting optimization.
- How deep it goes
- Defines squared-coefficient penalization, contrasts L1 behavior, and shows Ridge, logistic-regression, and XGBoost configurations without deriving the objective.
- Examples
- Uses a Ridge pipeline with polynomial features and an Optuna-tuned XGBoost regressor for Tokyo Airbnb MAE minimization.
- What you can do after
- Apply it via scikit-learn penalties and XGBoost lambda settings, evaluating candidate strengths with cross-validation.
Read how Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI explains weight decay →
Deep Learning with PyTorch, Second Edition
An optimization method that shrinks parameters in proportion to their current values, generally distinct from L2 regularization for adaptive optimizers.
- How it is used
- Uses weight decay as a regularization technique to limit parameter growth and help prevent overfitting.
- How deep it goes
- Explains the L2 penalty and its equivalence to proportional shrinkage under SGD, then introduces AdamW without deriving its update.
- Examples
- Demonstrates a PyTorch training loop with an explicit L2 penalty and constructs an AdamW optimizer for a segmentation model.
- What you can do after
- Apply it using PyTorch by adding an L2 penalty to the loss or selecting AdamW.
Read how Deep Learning with PyTorch, Second Edition explains weight decay →
Ensemble Methods for Machine Learning
L2 regularization penalizes the squared magnitude of model parameters or other model complexity measures in an optimization objective.
- How it is used
- Controls model complexity and overfitting by trading training fit against smaller parameters or simpler learners.
- How deep it goes
- Explains the squared-weight penalty, shrinkage, α tradeoffs, and library comparisons, but stops short of deriving optimization updates.
- Examples
- Uses correlated synthetic features, coefficient comparisons, test MSE and MAD, and regularized logistic-loss code.
- What you can do after
- Apply it via scikit-learn Ridge, compare L1 and elastic-net alternatives, and evaluate shrinkage using held-out regression metrics.
Read how Ensemble Methods for Machine Learning explains weight decay →
Deep Learning with R, Third Edition
L2 regularization is a regularization method that adds a penalty proportional to the squared values of model parameters.
- How it is used
- Reduces overfitting by constraining weight magnitudes and improving validation generalization.
- How deep it goes
- Defines the squared-weight penalty and demonstrates Keras configuration, but does not derive the optimization internals.
- Examples
- Uses an IMDb review classifier, Keras dense layers, validation-loss histories, and a regularizer_l2(0.002) setting.
- What you can do after
- Apply it via Keras by attaching kernel regularizers and comparing validation behavior during model tuning.
Read how Deep Learning with R, Third Edition explains weight decay →
Deep Learning with Python, Third Edition
- How it is used
- Use weight regularization to limit overfitting by discouraging large neural-network weights.
- How deep it goes
- Defines L2 cost as proportional to squared weights, equates it with weight decay, and demonstrates a Keras layer setting.
- Examples
- Uses IMDb review classification with Dense layers and an L2 coefficient of 0.002.
- What you can do after
- Apply L2 regularization through Keras kernel_regularizer settings while tuning for better generalization.
Read how Deep Learning with Python, Third Edition explains weight decay →
Build AI Drug Discovery Pipelines
L2 regularization penalizes the sum of squared model weights, shrinking them without generally setting them to zero.
- How it is used
- Controls coefficient magnitude to reduce overfitting in molecular machine-learning models with correlated features.
- How deep it goes
- Defines the squared-weight penalty and λ, then interprets validation curves but defers full diagnostics and hyperparameter-search details.
- Examples
- Uses molecular fingerprints, SVR validation curves, and VAE-CYC training with a 0.0001 learning rate.
- What you can do after
- Apply it via an SVR estimator by comparing validation performance across regularization strengths.
Read how Build AI Drug Discovery Pipelines explains weight decay →
Machine Learning Algorithms in Depth
A regularization method that adds a penalty proportional to the squared magnitude of model parameters to discourage large weights.
- How it is used
- Uses a parameter penalty to reduce overfitting in multilayer perceptrons.
- How deep it goes
- Defines the penalty’s purpose and parameter scope but stops short of deriving its update rule or implementation details.
- Examples
- Applies the penalty to nonbias weights W rather than biases b in each multilayer perceptron layer.
- What you can do after
- Apply it to nonbias weights when training a multilayer perceptron to discourage overfitting.
Read how Machine Learning Algorithms in Depth explains weight decay →
Causal AI
L2 regularization is a regularization method that penalizes the sum of squared parameter values.
- How it is used
- Steers parameter optimization toward smaller estimates by adding a squared-value penalty to the loss.
- How deep it goes
- Defines the penalty and its effect on large estimates, but does not derive an update rule or implementation procedure.
- Examples
- Uses neural-network estimation and negative log-likelihood as examples of optimization settings where regularization applies.
- What you can do after
- Apply L2 regularization by augmenting a loss with a sum-of-squares parameter penalty.
Timeless Algorithms, The Foundational Ideas
Weight decay is a regularization technique that penalizes large weights during model training.
- How it is used
- Constrains how a neural network uses its capacity during training to support generalization.
- How deep it goes
- Defines the loss penalty and explains its effect on reliance on large weights and generalization.
- What you can do after
- Understand how penalizing large weights trades some training fit for expected generalization to new observations.
Read how Timeless Algorithms, The Foundational Ideas explains weight decay →
Time Series Forecasting Using Foundation Models
L2 regularization is a penalty method that discourages large model weights by adding their squared magnitudes to the optimization objective.
- How it is used
- Uses ridge regression to incorporate exogenous features into TimesFM forecasts.
- How deep it goes
- Defines the L2 penalty and shows a ridge parameter, but stops short of deriving its optimization behavior.
- Examples
- Demonstrates the setting through a TimesFM covariate forecast configured with ridge=0.0.
- What you can do after
- Operate TimesFM's covariate forecast path by supplying a ridge setting alongside exogenous-feature inputs.
Read how Time Series Forecasting Using Foundation Models explains weight decay →
Also covered in
- Sutskever's List
- Grokking BayesIn early access (MEAP)
- Math and Architectures of Deep Learning
- AI-Powered Search
- Machines that ThinkIn early access (MEAP)
- Build a Text-to-Image Generator (from Scratch)
What to know first
- overfittingOverfitting is a modeling condition in which a model adapts too closely to particular data, including its noise or idiosyncrasies, and consequently generalizes poorly to new data.
- objective functionAn objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
- regularizationRegularization is a modeling technique that constrains learning or penalizes model complexity to discourage overfitting and improve generalization.
- linear regressionLinear regression is a statistical method that models a numeric response as a linear function of one or more predictor variables with fitted parameters.
Related concepts
- regularizationkind of · Regularization is a modeling technique that constrains learning or penalizes model complexity to discourage overfitting and improve generalization.
- overfittingaddresses · Overfitting is a modeling condition in which a model adapts too closely to particular data, including its noise or idiosyncrasies, and consequently generalizes poorly to new data.
- l2 normuses · The L2 norm of a vector is its Euclidean length, calculated as the square root of the sum of the squares of its components.
- ridge regressionused by · Ridge regression is linear regression fitted with an L2 penalty on coefficient magnitudes, which shrinks estimates toward zero.
- AlexNetused by · AlexNet is a convolutional neural network architecture designed for image classification and recognized for its landmark performance in large-scale image recognition.
Questions about weight decay
What is weight decay?
A regularization technique that reduces model parameter magnitudes during optimization, either by adding a penalty to the objective or by directly shrinking parameters.
Which Manning books explain weight decay?
Grokking Machine Learning, Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI, Deep Learning with PyTorch, Second Edition, Ensemble Methods for Machine Learning, Deep Learning with R, Third Edition, Deep Learning with Python, Third Edition, Build AI Drug Discovery Pipelines, Machine Learning Algorithms in Depth and 3 more.
What should I know before learning weight decay?
Manning books that teach weight decay usually introduce overfitting, objective function, regularization and linear regression first.
What is weight decay a kind of?
Weight decay is a kind of regularization. Regularization is a modeling technique that constrains learning or penalizes model complexity to discourage overfitting and improve generalization.
What is weight decay also called?
Weight decay is also called L2 regularization or l2 penalty.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.










