Regularization

As explained in Sutskever's List

A regularizer is a modeling mechanism that discourages undesirable solutions to improve generalization or control model complexity.

Regularization discourages solutions that memorize training examples, aiming to preserve generalization. The classical account says training error falls as model capacity grows, while test error eventually rises; regularization limits capacity or improves robustness, favoring the bottom of the bias–variance curve. section 8.5.2

AlexNet and overfitting

AlexNet’s 60 million parameters raised concerns about overfitting. Its authors used data augmentation, dropout, and weight decay. Dropout randomly disables neurons during training; AlexNet dropped half the neurons in its first two fully connected layers. section 8.5.2 section 2.5.2

Modern overparameterized networks complicate the classical picture: test error can fall again beyond the interpolation threshold. Explicit regularizers and an optimizer’s implicit biases can steer models that fit every training example toward simpler functions. In grokking, sufficient regularization accompanies a rise and then a fall in internal complexity as the network shifts from memorization to generalization. section 8.5.2 section 8.5

Figure 8.9 Left: the classical bias–variance tradeoff. Training error (called “risk” in this image) is shown as a dashed line. It falls monotonically, whereas the test error (solid line) is U-shaped, with the bottom being optimal. Right: beyond the interpolation threshold, test error peaks and then declines again in the overparameterized (interpolating) regime, while training risk remains near zero. Used with permission granted by the lead author (Mikhail Belkin).

Compare the classical U-shaped test-error curve with the renewed decline beyond the interpolation threshold. section 8.5.2

How Sutskever's List covers it

How deep it goes

Explains classical capacity control, dropout’s training mechanism, and modern overparameterized generalization, but gives no formal derivation or tuning procedure.

Examples

Uses polynomial fitting, AlexNet, and modular-arithmetic grokking to contrast memorization with generalization.

What you can do after

Evaluate how explicit regularizers and optimizer biases relate to generalization, without assuming larger capacity necessarily worsens test error.

How it connects

Links regularization to overfitting, generalization, the bias–variance tradeoff, dropout, weight decay, data augmentation, and optimizer implicit bias.

liveBookmanning.com

Read Sutskever's List on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningA regularization function is a function that assigns a penalty to model parameters or complexity as part of a regularized objective.
• 

  liveBookmanning.com
  Grokking Machine LearningRegularization is a model-training technique that penalizes complexity while optimizing predictive performance.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionRegularization is a modeling technique that constrains learning to reduce overfitting and improve generalization to unseen data.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIRegularization is a technique that limits model complexity by penalizing parameter values to reduce overfitting and improve generalization.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningThe practice of constraining a learned representation or model to encourage desirable structure such as smoothness or compactness.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA method that constrains a model during training to reduce overfitting and improve generalization.
• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackRegularization is a technique that constrains optimization to reduce undesirable or excessive fitting to an objective.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA set of techniques that reduce overfitting and improve generalization by constraining model behavior during training.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthA modeling technique that adds a penalty or constraint to discourage undesirable parameter values, such as excessive magnitude.
• 

  liveBookmanning.com
  Bayesian Optimization in ActionA loss function quantifies the discrepancy between a model's predictions and desired outcomes or data, with lower values indicating better fit.
• 

  liveBookmanning.com
  Causal AIRegularization is a modeling technique that adds a penalty or constraint to an optimization objective to discourage undesirable parameter values.
• 

  liveBookmanning.com
  Machine Learning System DesignRegularization is a technique that constrains model fitting to discourage overly complex solutions and improve generalization.

Compare how 13 books explain regularization →

Related concepts

• overfittingaddresses · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.
• dropoutincludes · Dropout is a neural-network regularization technique that randomly suppresses units, connections, or activations during training to reduce overfitting.
• regularization parameterconfigured by · A regularization parameter is a model-fitting setting that controls the influence of a penalty or constraint relative to fit to the data.
• variational autoencoderused by · A variational autoencoder is a generative neural network that learns a probabilistic latent representation of data and decodes samples from it to reconstruct or generate data.
• Kullback–Leibler divergenceuses · Kullback–Leibler divergence is a nonnegative, directed measure of the information lost when one probability distribution approximates another, computed as an expected log ratio under the first distribution.
• marginal likelihoodequivalent to · Marginal likelihood, or Bayesian evidence, is the probability or density of observed data under a model after marginalizing over unknown parameters or latent variables according to their prior distributions.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.