Weight decay

As explained in Ensemble Methods for Machine Learning

L2 regularization penalizes the squared magnitude of model parameters or other model complexity measures in an optimization objective.

Weight decay, commonly implemented as L2 regularization, adds a penalty for large parameter magnitudes to the learning objective. Training therefore balances fitting the data against keeping the model simpler: increasing α strengthens shrinkage, while excessive regularization can underfit. section 7.1.1 section 6.2.2

The practical effect is easiest to see by contrasting related regression methods:

• Ridge regression minimizes squared error plus the sum of squared weights, shrinking coefficients toward zero without generally making them exactly zero. section 7.1.1
• LASSO uses an L1 penalty instead, encouraging sparse models and implicit feature selection. section 7.1.1
• Elastic net combines L1 and L2 penalties to balance sparsity with robustness to correlated features. section 7.1.1
• In tree ensembles, regularized objectives can combine L2 regularization with a penalty on the number of leaf nodes. section 6.6

Correlated regression features

A synthetic regression dataset contains two highly correlated features, with the second approximately three times the first. Ordinary least squares learns coefficients of about [-1.46, -0.32], whereas ridge with α = 0.5 learns about [-0.34, -0.70]. The ridge model slightly improves test MSE while redistributing and shrinking the coefficients. section 7.1.1

Figure 7.1 Data for a synthetic regression problem to which we fit several linear regression models, generated by the univariate (1D), noisy function f(x) = -2.5x + 3.2 shown by the line overlaid on the data points.

Inspect the noisy synthetic regression data used to compare unregularized and regularized linear models. section 7.1.1

How Ensemble Methods for Machine Learning covers it

How deep it goes

Explains the squared-weight penalty, shrinkage, α tradeoffs, and library comparisons, but stops short of deriving optimization updates.

Examples

Uses correlated synthetic features, coefficient comparisons, test MSE and MAD, and regularized logistic-loss code.

What it assumes

Assumes familiarity with learning objectives, model weights, loss functions, regression, and train-test evaluation.

What you can do after

Apply it via scikit-learn Ridge, compare L1 and elastic-net alternatives, and evaluate shrinkage using held-out regression metrics.

How it connects

Connects ridge regression, L1 regularization, elastic net, multicollinearity, gradient boosting, logistic regression, and XGBoost.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningRidge regression is a linear regression technique that uses an L2 coefficient penalty, shrinking coefficients toward zero without usually making them zero.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIL2 regularization adds a penalty proportional to the squared values of model parameters to constrain model complexity.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionAn optimization method that shrinks parameters in proportion to their current values, generally distinct from L2 regularization for adaptive optimizers.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionL2 regularization is a regularization method that adds a penalty proportional to the squared values of model parameters.
• 

  liveBookmanning.com
  Deep Learning with Python, Third Edition
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthA regularization method that adds a penalty proportional to the squared magnitude of model parameters to discourage large weights.
• 

  liveBookmanning.com
  Causal AIL2 regularization is a regularization method that penalizes the sum of squared parameter values.
• 

  liveBookmanning.com
  Time Series Forecasting Using Foundation ModelsL2 regularization is a penalty method that discourages large model weights by adding their squared magnitudes to the optimization objective.

Compare how 9 books explain weight decay →

Related concepts

• regularizationkind of · Regularization is a modeling technique that constrains or penalizes solutions during fitting or optimization to discourage undesirable complexity or behavior and improve generalization or estimation.
• overfittingaddresses · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.
• ridge regressionused by · Ridge regression is linear regression that applies an L2 penalty to coefficient magnitudes, shrinking estimates toward zero to reduce overfitting.
• l2 normuses · The L2 norm is the nonnegative Euclidean magnitude of a vector, equal to the square root of the sum of its components’ squares, and to absolute value for a scalar.
• AlexNetused by · AlexNet is a convolutional neural network architecture designed for image classification and recognized for its landmark performance in large-scale image recognition.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.