Weight decay

As explained in Grokking Machine Learning

Ridge regression is a linear regression technique that uses an L2 coefficient penalty, shrinking coefficients toward zero without usually making them zero.

Weight decay, also called L2 regularization or ridge regression, discourages overly complex regression models by adding a penalty based on the L2 norm of their coefficients to the regression error. A regularization parameter controls how much that penalty matters. section 4.5

Figure 4.7 Both L1 and L2 shrink the size of the coefficient. L1 regularization (left) does it much faster, because it subtracts a fixed amount, so it is likely to eventually become zero. L2 regularization takes much longer, because it multiplies the coefficient by a small factor, so it never reaches zero.

The comparison illustrates how L2 regularization gradually shrinks coefficients, unlike L1 regularization, which can drive them to zero. section 4.5

For a polynomial regression example, compare models with no regularization, L1 regularization, and L2 regularization. The unregularized model fits training points closely but performs poorly on testing points; the L2 model fits both sets well. In Turi Create, set `l2_penalty` when creating the model. section 4.6

How Grokking Machine Learning covers it

How deep it goes

Defines the L2-penalized error and illustrates gradual coefficient shrinkage with a polynomial regression comparison.

Examples

Compares unregularized, L1-regularized, and L2-regularized polynomial models on training and testing data.

What you can do after

Apply via Turi Create by setting `l2_penalty` and comparing model performance on training and testing data.

How it connects

Connects ridge regression with L2 regularization, L1 regularization, polynomial regression, and overfitting.

liveBookmanning.com

Read Grokking Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIL2 regularization adds a penalty proportional to the squared values of model parameters to constrain model complexity.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionAn optimization method that shrinks parameters in proportion to their current values, generally distinct from L2 regularization for adaptive optimizers.
• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningL2 regularization penalizes the squared magnitude of model parameters or other model complexity measures in an optimization objective.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionL2 regularization is a regularization method that adds a penalty proportional to the squared values of model parameters.
• 

  liveBookmanning.com
  Deep Learning with Python, Third Edition
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthA regularization method that adds a penalty proportional to the squared magnitude of model parameters to discourage large weights.
• 

  liveBookmanning.com
  Causal AIL2 regularization is a regularization method that penalizes the sum of squared parameter values.
• 

  liveBookmanning.com
  Time Series Forecasting Using Foundation ModelsL2 regularization is a penalty method that discourages large model weights by adding their squared magnitudes to the optimization objective.

Compare how 9 books explain weight decay →

Related concepts

• regularizationkind of · Regularization is a modeling technique that constrains or penalizes solutions during fitting or optimization to discourage undesirable complexity or behavior and improve generalization or estimation.
• overfittingaddresses · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.
• ridge regressionused by · Ridge regression is linear regression that applies an L2 penalty to coefficient magnitudes, shrinking estimates toward zero to reduce overfitting.
• l2 normuses · The L2 norm is the nonnegative Euclidean magnitude of a vector, equal to the square root of the sum of its components’ squares, and to absolute value for a scalar.
• AlexNetused by · AlexNet is a convolutional neural network architecture designed for image classification and recognized for its landmark performance in large-scale image recognition.

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning. The section links point to the parts of the book it draws on.