liveBook · Manning

Regularization

As explained in Grokking Bayes

Regularization is a modeling technique that constrains or penalizes parameter configurations to reduce overfitting and control model complexity.

Regularization keeps a model from chasing noisy data. It balances prediction errors against model complexity, favoring simpler parameter settings that are less likely to overfit. section 3.16 section 7.2

In linear regression, regularization penalizes large weights. A model may use many features—such as a house’s size, location, and age—but discourages any one noisy feature from dominating the prediction. Minimizing prediction error and weight magnitude together produces a conservative fit. section 3.16

Bayesian models express the same preference through priors:

  • A strong, zero-centered normal prior concentrates weight values near zero, reducing the influence of noisy features. section 3.16
  • This produces the same effect as L₂ regularization in ordinary linear regression: both balance data fit against small weights. section 3.16
  • With limited, noisy data, the prior prevents the data alone from pushing the model toward implausible conclusions. section 3.16

Sharing information across schools

Suppose separate school models estimate the effect of coaching on test scores. Pooling information across schools regularizes each school’s predictive distribution, reducing excessive variance and making extreme effects less likely. section 7.2

How Grokking Bayes covers it

How deep it goes
Explains weight penalties, prior-based shrinkage, L₂ equivalence, and hierarchical pooling without deriving optimization or posterior calculations.
Examples
Uses house-price features, zero-centered weights, and pooled school coaching effects to illustrate shrinkage and variance reduction.
What it assumes
Assumes familiarity with regression weights and Bayesian priors while motivating both concepts through noisy prediction examples.
What you can do after
Evaluate/understand how penalties, priors, and pooling produce simpler and more conservative predictive models.
How it connects
Connects regularization with Bayesian priors, Bayesian linear regression, L₂ regularization, overfitting, hierarchical models, and pooling.

Also explained in

Compare how 17 books explain regularization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Bayes. The section links point to the parts of the book it draws on.