liveBook · Manning

Regularization

As explained in Grokking Machine Learning, Second Edition

Regularization is a modeling technique that constrains parameter values or model complexity to reduce overfitting.

Regularization helps a model avoid overfitting by balancing two goals: fit the data well and stay simple. Instead of minimizing prediction error alone, training also includes a complexity penalty. Think of repairing a leaking roof: a bandage is too simple, titanium is excessive, and a practical repair balances results with cost. section 4.7 section 4.10

Error = Regression error + Regularization term

The modified error combines regression error, which measures fit, with a regularization term, which measures model complexity. section 4.10

The regularization parameter, λ, controls the trade-off. At λ = 0, the complexity penalty disappears. A larger λ pushes training toward a simpler model, possibly one that fits the data less well. Try candidate values on a validation set and keep the one that performs best there. section 4.11

Two common penalties shrink model coefficients in different ways:

  • L1 regularization uses the L1 norm and produces lasso regression. It subtracts a fixed amount during shrinking, so coefficients can reach zero; this is useful when many features should be turned off. section 4.13 section 4.15
  • L2 regularization uses the L2 norm and produces ridge regression. It shrinks coefficients without making them zero, which is recommended when a dataset has few features. section 4.13 section 4.15
Figure 4.7 Both L1 and L2 shrink the size of the coefficient. L1 regularization (left) does it much faster, because it subtracts a fixed amount, so it is likely to eventually become zero. L2 regularization takes much longer, because it multiplies the coefficient by a small factor, so it never reaches zero.

This comparison makes the difference visible: L1 steadily reaches zero, while L2 reduces a coefficient by a factor and leaves it nonzero. section 4.13

Taming a high-degree curve

For downward-opening data, expanding one feature into powers up to degree 20 creates a flexible polynomial model with many columns. Train an unregularized model alongside Lasso and Ridge models using alpha=0.01, then compare their curves on held-out test data. Regularization provides a way to tune that flexible model rather than accepting its full complexity. section 4.14

The same idea applies to neural networks: add a penalty for high weights to the error function, excluding biases. L1 uses absolute weight values; L2 uses squared values. Regularization can be paired with dropout as another technique for reducing overfitting. section 10.13

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Explains modified errors, λ tuning, L1/L2 coefficient shrinking, and scikit-learn use; formal gradient-descent mathematics is deferred to an appendix.
Examples
Uses roof repairs, movie recommendations, coefficient shrinking, and degree-20 polynomial fitting with Lasso and Ridge.
What it assumes
Builds on regression error, coefficients, polynomial regression, and training/testing splits while introducing regularization from an intuitive analogy.
What you can do after
Apply it via scikit-learn Lasso or Ridge by selecting alpha and comparing regularized polynomial models on held-out data.
How it connects
Connects regression error, L1 norm, L2 norm, lasso regression, ridge regression, validation, overfitting, polynomial regression, neural networks, and dropout.

Also explained in

Compare how 17 books explain regularization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.