Regularization

As explained in Grokking Machine Learning

Regularization is a model-training technique that penalizes complexity while optimizing predictive performance.

Suppose three roof repairs stop a leak: a bandage, shingles, and a block of titanium. Measuring only water left leaking favors titanium, but its cost makes shingles the more sensible choice. A model can face the same problem: a complicated solution may fit its training data well without being the best choice for new data. Regularization encourages a balance between fit and complexity while the model is trained. section 4.5 section 4.0

The idea is to add a complexity penalty to the error being minimized: error = regression error + λ × regularization term. The parameter λ controls how much the penalty counts. Unlike selecting among several models by a complexity graph, this penalty is part of training a model. It does not itself require a testing set, although testing remains useful for evaluating predictions. section 4.5 section 4.6

Two common ways to measure complexity are:

• L1 regularization, or lasso regression, uses the L1 norm. It can shrink many feature coefficients to zero, making it useful when there are numerous features. section 4.5 section 4.7
• L2 regularization, or ridge regression, uses the L2 norm. It tends to make coefficients small rather than zero; it is recommended here when there are fewer features. section 4.5 section 4.7

Try it with a curved dataset

For data shaped like a downward-opening parabola, add columns for powers of x and fit polynomial regression. The example expands the data to 200 columns, splits it into training and testing sets, then trains three Turi Create models: no penalty, L1 penalty 0.1, and L2 penalty 0.1. Compare their curves, test predictions, and coefficients. The unregularized model has large coefficients; the penalized models have smaller ones. What changes when you adjust the penalties? Keep the testing data out of training and model decisions. section 4.6 section 4.7

Figure 4.9 Three polynomial regression models for our dataset. The model on the left has no regularization, the model in the middle has L1 regularization with a parameter of 0.1, and the model on the right has L2 regularization with a parameter of 0.1.

Compare the fitted curves for no regularization, L1 regularization, and L2 regularization on the same dataset. section 4.6

How Grokking Machine Learning covers it

How deep it goes

Develops a roof-repair analogy, a penalized error expression, coefficient-shrinkage differences, and a Turi Create experiment.

Examples

Compares three roof repairs and three polynomial fits using no penalty, L1 regularization, and L2 regularization.

What it assumes

Builds from an everyday cost analogy and explains polynomial feature expansion before using regression models in Turi Create.

What you can do after

Apply it via Turi Create’s linear_regression.create with l1_penalty or l2_penalty, then inspect test predictions and coefficients.

How it connects

Connects regularization to overfitting, underfitting, polynomial regression, L1 and L2 norms, testing, and validation.

liveBookmanning.com

Read Grokking Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningA regularization function is a function that assigns a penalty to model parameters or complexity as part of a regularized objective.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionRegularization is a modeling technique that constrains learning to reduce overfitting and improve generalization to unseen data.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIRegularization is a technique that limits model complexity by penalizing parameter values to reduce overfitting and improve generalization.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningThe practice of constraining a learned representation or model to encourage desirable structure such as smoothness or compactness.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA method that constrains a model during training to reduce overfitting and improve generalization.
• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackRegularization is a technique that constrains optimization to reduce undesirable or excessive fitting to an objective.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA set of techniques that reduce overfitting and improve generalization by constraining model behavior during training.
• 

  liveBookmanning.com
  Sutskever's ListA regularizer is a modeling mechanism that discourages undesirable solutions to improve generalization or control model complexity.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthA modeling technique that adds a penalty or constraint to discourage undesirable parameter values, such as excessive magnitude.
• 

  liveBookmanning.com
  Bayesian Optimization in ActionA loss function quantifies the discrepancy between a model's predictions and desired outcomes or data, with lower values indicating better fit.
• 

  liveBookmanning.com
  Causal AIRegularization is a modeling technique that adds a penalty or constraint to an optimization objective to discourage undesirable parameter values.
• 

  liveBookmanning.com
  Machine Learning System DesignRegularization is a technique that constrains model fitting to discourage overly complex solutions and improve generalization.

Compare how 13 books explain regularization →

Related concepts

• overfittingaddresses · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.
• dropoutincludes · Dropout is a neural-network regularization technique that randomly suppresses units, connections, or activations during training to reduce overfitting.
• regularization parameterconfigured by · A regularization parameter is a model-fitting setting that controls the influence of a penalty or constraint relative to fit to the data.
• variational autoencoderused by · A variational autoencoder is a generative neural network that learns a probabilistic latent representation of data and decodes samples from it to reconstruct or generate data.
• Kullback–Leibler divergenceuses · Kullback–Leibler divergence is a nonnegative, directed measure of the information lost when one probability distribution approximates another, computed as an expected log ratio under the first distribution.
• marginal likelihoodequivalent to · Marginal likelihood, or Bayesian evidence, is the probability or density of observed data under a model after marginalizing over unknown parameters or latent variables according to their prior distributions.

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning. The section links point to the parts of the book it draws on.