Regularization

As explained in Machine Learning System Design

Regularization is a technique that constrains model fitting to discourage overly complex solutions and improve generalization.

Regularization constrains model fitting to discourage overly complex solutions and improve generalization. It becomes relevant when training loss falls while validation loss rises, signaling overfitting caused by high variance. section 9.1.3

Use the loss curves to guide the response:

• Confirm that the pattern is genuine rather than caused by an implementation problem, such as incorrect normalization. section 9.1.3
• Restrict model capacity by reducing model complexity directly or increasing regularization. section 9.1.3

Figure 9.6 Loss is decreasing for training but not validation, reflecting potential overfit.

Compare training and validation loss curves to recognize the visual signature of potential overfitting. section 9.1.3

How Machine Learning System Design covers it

How deep it goes

Explains the overfitting pattern and recommends capacity reduction or stronger regularization, without detailing specific methods.

Examples

Uses diverging training and validation loss curves, plus normalization errors as a diagnostic contrast.

What it assumes

Assumes familiarity with training loss, validation loss, model complexity, and basic loss-curve interpretation.

What you can do after

Diagnose overfitting from loss curves and choose between reducing complexity and increasing regularization.

How it connects

Connects regularization with overfitting, high variance, model capacity, training loss, and validation loss.

liveBookmanning.com

Read Machine Learning System Design on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningA regularization function is a function that assigns a penalty to model parameters or complexity as part of a regularized objective.
• 

  liveBookmanning.com
  Grokking Machine LearningRegularization is a model-training technique that penalizes complexity while optimizing predictive performance.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionRegularization is a modeling technique that constrains learning to reduce overfitting and improve generalization to unseen data.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIRegularization is a technique that limits model complexity by penalizing parameter values to reduce overfitting and improve generalization.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningThe practice of constraining a learned representation or model to encourage desirable structure such as smoothness or compactness.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA method that constrains a model during training to reduce overfitting and improve generalization.
• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackRegularization is a technique that constrains optimization to reduce undesirable or excessive fitting to an objective.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA set of techniques that reduce overfitting and improve generalization by constraining model behavior during training.
• 

  liveBookmanning.com
  Sutskever's ListA regularizer is a modeling mechanism that discourages undesirable solutions to improve generalization or control model complexity.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthA modeling technique that adds a penalty or constraint to discourage undesirable parameter values, such as excessive magnitude.
• 

  liveBookmanning.com
  Bayesian Optimization in ActionA loss function quantifies the discrepancy between a model's predictions and desired outcomes or data, with lower values indicating better fit.
• 

  liveBookmanning.com
  Causal AIRegularization is a modeling technique that adds a penalty or constraint to an optimization objective to discourage undesirable parameter values.

Compare how 13 books explain regularization →

Related concepts

• overfittingaddresses · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.
• dropoutincludes · Dropout is a neural-network regularization technique that randomly suppresses units, connections, or activations during training to reduce overfitting.
• regularization parameterconfigured by · A regularization parameter is a model-fitting setting that controls the influence of a penalty or constraint relative to fit to the data.
• variational autoencoderused by · A variational autoencoder is a generative neural network that learns a probabilistic latent representation of data and decodes samples from it to reconstruct or generate data.
• Kullback–Leibler divergenceuses · Kullback–Leibler divergence is a nonnegative, directed measure of the information lost when one probability distribution approximates another, computed as an expected log ratio under the first distribution.
• marginal likelihoodequivalent to · Marginal likelihood, or Bayesian evidence, is the probability or density of observed data under a model after marginalizing over unknown parameters or latent variables according to their prior distributions.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.