Regularization helps a model avoid overfitting by balancing two goals: fit the data well and stay simple. Instead of minimizing prediction error alone, training also includes a complexity penalty. Think of repairing a leaking roof: a bandage is too simple, titanium is excessive, and a practical repair balances results with cost. section 4.7 section 4.10
The modified error combines regression error, which measures fit, with a regularization term, which measures model complexity. section 4.10
The regularization parameter, λ, controls the trade-off. At λ = 0, the complexity penalty disappears. A larger λ pushes training toward a simpler model, possibly one that fits the data less well. Try candidate values on a validation set and keep the one that performs best there. section 4.11
Two common penalties shrink model coefficients in different ways:
- L1 regularization uses the L1 norm and produces lasso regression. It subtracts a fixed amount during shrinking, so coefficients can reach zero; this is useful when many features should be turned off. section 4.13 section 4.15
- L2 regularization uses the L2 norm and produces ridge regression. It shrinks coefficients without making them zero, which is recommended when a dataset has few features. section 4.13 section 4.15
Figure 4.7 Both L1 and L2 shrink the size of the coefficient. L1 regularization (left) does it much faster, because it subtracts a fixed amount, so it is likely to eventually become zero. L2 regularization takes much longer, because it multiplies the coefficient by a small factor, so it never reaches zero.
This comparison makes the difference visible: L1 steadily reaches zero, while L2 reduces a coefficient by a factor and leaves it nonzero. section 4.13
Taming a high-degree curve
For downward-opening data, expanding one feature into powers up to degree 20 creates a flexible polynomial model with many columns. Train an unregularized model alongside Lasso and Ridge models using alpha=0.01, then compare their curves on held-out test data. Regularization provides a way to tune that flexible model rather than accepting its full complexity. section 4.14
The same idea applies to neural networks: add a penalty for high weights to the error function, excluding biases. L1 uses absolute weight values; L2 uses squared values. Regularization can be paired with dropout as another technique for reducing overfitting. section 10.13
















