liveBook · Manning

Ridge regression

As explained in Grokking Machine Learning, Second Edition

L2 regularization penalizes squared coefficient values, shrinking them toward zero without usually making them exactly zero.

Ridge regression is linear regression with an extra cost for large coefficients. Along with fitting the data, training tries to keep the model’s weights small. This L2 regularization can make a flexible model less likely to chase every wiggle in its training data. section 4.10 section 4.15

Ridge regression error = Regression error + L2 norm

The ridge objective adds an L2 complexity term to ordinary regression error, so a good model must balance fit with smaller coefficients. section 4.10

The key effect is shrinkage: ridge makes all coefficients smaller, but rarely makes any coefficient exactly zero. That makes it useful when there are only a few features and each may matter. By contrast, lasso regression uses L1 regularization and can remove features by pushing coefficients to zero. section 4.12 section 4.15

Keeping all movie signals, but quieter

For a movie-watch-time prediction with 10 input movies, an unregularized model can have large positive and negative coefficients. Ridge retraining produces smaller coefficients across all 10 inputs, such as changing 22 to 0.2 and 378 to 2.9, while retaining the inputs rather than eliminating most of them. section 4.12

One intuitive training picture is to repeatedly improve a prediction and then shrink the coefficients. With L2 shrinkage, a coefficient such as 2 is repeatedly multiplied by a value slightly below 1: it falls through values like 1.98 and 1.9602, approaching zero without reaching it. section 4.13

Figure 4.7 Both L1 and L2 shrink the size of the coefficient. L1 regularization (left) does it much faster, because it subtracts a fixed amount, so it is likely to eventually become zero. L2 regularization takes much longer, because it multiplies the coefficient by a small factor, so it never reaches zero.

This comparison makes ridge’s gradual multiplicative shrinkage visible beside lasso’s faster route to exact zero. section 4.13

In scikit-learn, create `Ridge(alpha=0.01)` and fit it like another regression model. The `alpha` value controls the regularization penalty; zero means no regularization. On a degree-20 polynomial fit to parabolic data, ridge with alpha 0.01 achieved testing RMSE 0.104, compared with 1862.044 without regularization. section 4.14

Figure 4.9 Three polynomial regression models for our dataset. The model on the left has no regularization, the model in the middle has L1 regularization with a parameter of 0.01, and the model on the right has L2 regularization with a parameter of 0.01.

The three fitted curves compare an unregularized polynomial with L1- and L2-regularized alternatives on training and testing points. section 4.14

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Explains the L2 objective, coefficient shrinkage, iterative intuition, and scikit-learn use; defers the mathematical justification.
Examples
Uses movie watch-time coefficients and a degree-20 polynomial fitted to parabolic data.
What it assumes
Builds from regression error and coefficients while introducing L1 and L2 regularization in plain terms.
What you can do after
Apply it via scikit-learn’s Ridge class, choose alpha, fit a model, and compare testing RMSE.
How it connects
Connects ridge regression to linear regression, L2 regularization, lasso regression, polynomial regression, testing, and overfitting.

Also explained in

Compare how 4 books explain ridge regression →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.