Weight decay
Weight decay is a model-training regularization technique that discourages large parameter values by penalizing their magnitudes or shrinking them during optimization.
Grokking Machine Learning
Ridge regression is a linear regression technique that uses an L2 coefficient penalty, shrinking coefficients toward zero without usually making them zero.
- How it is used
- Reduces overfitting by adding a coefficient-complexity penalty to the regression error during training.
- How deep it goes
- Defines the L2-penalized error and illustrates gradual coefficient shrinkage with a polynomial regression comparison.
- Examples
- Compares unregularized, L1-regularized, and L2-regularized polynomial models on training and testing data.
- What you can do after
- Apply via Turi Create by setting `l2_penalty` and comparing model performance on training and testing data.
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
L2 regularization adds a penalty proportional to the squared values of model parameters to constrain model complexity.
- How it is used
- Reduces overfitting and model complexity in regression, logistic regression, and gradient-boosting optimization.
- How deep it goes
- Defines squared-coefficient penalization, contrasts L1 behavior, and shows Ridge, logistic-regression, and XGBoost configurations without deriving the objective.
- Examples
- Uses a Ridge pipeline with polynomial features and an Optuna-tuned XGBoost regressor for Tokyo Airbnb MAE minimization.
- What you can do after
- Apply it via scikit-learn penalties and XGBoost lambda settings, evaluating candidate strengths with cross-validation.
Read how Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI explains weight decay →
Deep Learning with PyTorch, Second Edition
An optimization method that shrinks parameters in proportion to their current values, generally distinct from L2 regularization for adaptive optimizers.
- How it is used
- Limits parameter growth to help prevent overfitting in convolutional models and also appears in segmentation training.
- How deep it goes
- Works through an L2 loss penalty and its SGD shrinkage intuition, but does not explain AdamW’s weight-decay update.
- Examples
- Uses a PyTorch training loop with an L2 multiplier of 0.001 and a segmentation optimizer initialized at learning rate 0.00006.
- What you can do after
- Apply it via PyTorch by adding an L2 penalty to training loss or selecting AdamW for a segmentation model.
Read how Deep Learning with PyTorch, Second Edition explains weight decay →
Ensemble Methods for Machine Learning
L2 regularization penalizes the squared magnitude of model parameters or other model complexity measures in an optimization objective.
- How it is used
- Controls model complexity and overfitting by trading training fit against smaller parameters or simpler learners.
- How deep it goes
- Explains the squared-weight penalty, shrinkage, α tradeoffs, and library comparisons, but stops short of deriving optimization updates.
- Examples
- Uses correlated synthetic features, coefficient comparisons, test MSE and MAD, and regularized logistic-loss code.
- What you can do after
- Apply it via scikit-learn Ridge, compare L1 and elastic-net alternatives, and evaluate shrinkage using held-out regression metrics.
Read how Ensemble Methods for Machine Learning explains weight decay →
Deep Learning with R, Third Edition
L2 regularization is a regularization method that adds a penalty proportional to the squared values of model parameters.
- How it is used
- Reduces overfitting by constraining weight magnitudes and improving validation generalization.
- How deep it goes
- Defines the squared-weight penalty and demonstrates Keras configuration, but does not derive the optimization internals.
- Examples
- Uses an IMDb review classifier, Keras dense layers, validation-loss histories, and a regularizer_l2(0.002) setting.
- What you can do after
- Apply it via Keras by attaching kernel regularizers and comparing validation behavior during model tuning.
Read how Deep Learning with R, Third Edition explains weight decay →
Deep Learning with Python, Third Edition
- How it is used
- Reduces overfitting by penalizing large weights as a form of model regularization.
- How deep it goes
- Defines L1 and L2 penalties, equates weight decay with L2, and demonstrates Keras usage without deriving optimizer internals.
- Examples
- Uses an IMDb review classifier with L2 penalties on two Dense-layer kernels.
- What you can do after
- Apply it via Keras by attaching an L2 regularizer to Dense-layer kernels.
Read how Deep Learning with Python, Third Edition explains weight decay →
Machine Learning Algorithms in Depth
A regularization method that adds a penalty proportional to the squared magnitude of model parameters to discourage large weights.
- How it is used
- Uses a parameter penalty to reduce overfitting in multilayer perceptrons.
- How deep it goes
- Defines the penalty’s purpose and parameter scope but stops short of deriving its update rule or implementation details.
- Examples
- Applies the penalty to nonbias weights W rather than biases b in each multilayer perceptron layer.
- What you can do after
- Apply it to nonbias weights when training a multilayer perceptron to discourage overfitting.
Read how Machine Learning Algorithms in Depth explains weight decay →
Causal AI
L2 regularization is a regularization method that penalizes the sum of squared parameter values.
- How it is used
- Steers parameter optimization toward smaller estimates by adding a squared-value penalty to the loss.
- How deep it goes
- Defines the penalty and its effect on large estimates, but does not derive an update rule or implementation procedure.
- Examples
- Uses neural-network estimation and negative log-likelihood as examples of optimization settings where regularization applies.
- What you can do after
- Apply L2 regularization by augmenting a loss with a sum-of-squares parameter penalty.
Time Series Forecasting Using Foundation Models
L2 regularization is a penalty method that discourages large model weights by adding their squared magnitudes to the optimization objective.
- How it is used
- Uses ridge regression to incorporate exogenous features into TimesFM forecasts.
- How deep it goes
- Defines the L2 penalty and shows a ridge parameter, but stops short of deriving its optimization behavior.
- Examples
- Demonstrates the setting through a TimesFM covariate forecast configured with ridge=0.0.
- What you can do after
- Operate TimesFM's covariate forecast path by supplying a ridge setting alongside exogenous-feature inputs.
Read how Time Series Forecasting Using Foundation Models explains weight decay →
Related concepts
- regularizationkind of · Regularization is a modeling technique that constrains or penalizes solutions during fitting or optimization to discourage undesirable complexity or behavior and improve generalization or estimation.
- overfittingaddresses · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.
- ridge regressionused by · Ridge regression is linear regression that applies an L2 penalty to coefficient magnitudes, shrinking estimates toward zero to reduce overfitting.
- l2 normuses · The L2 norm is the nonnegative Euclidean magnitude of a vector, equal to the square root of the sum of its components’ squares, and to absolute value for a scalar.
- AlexNetused by · AlexNet is a convolutional neural network architecture designed for image classification and recognized for its landmark performance in large-scale image recognition.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.