Regularization

As explained in Machine Learning Algorithms in Depth

A modeling technique that adds a penalty or constraint to discourage undesirable parameter values, such as excessive magnitude.

Regularization modifies a loss function to prefer smaller parameter values, resolving cases where several parameter settings achieve the same empirical risk. section 5.3

For a support vector machine, the penalty applies to θ but not θ₀: θ controls boundary orientation, while θ₀ controls its offset. Minimizing the L2 norm of θ under correct-classification constraints increases the margin and yields a maximum-margin solution. section 5.3

The treatment connects regularization to:

• Empirical risk minimization, by replacing an underdetermined loss objective with a preference for smaller parameters. section 5.3
• Quadratic programming, by expressing margin maximization as a quadratic objective with linear classification constraints. section 5.3
• Logistic regression, where regularization is added to the objective and its gradient before stochastic-gradient updates. section 5.4

Implementation examples

An SVM implementation exposes a regularization parameter C alongside kernel settings. An SGD logistic-regression implementation initializes lmbda and computes a regularized objective and gradient while training on a synthetic Gaussian-mixture dataset.

Figure 5.8 Max margin solution of SVM classifier

Visualizes how shrinking the parameter norm pushes margin boundaries apart until the unique maximum-margin solution is reached. section 5.3

How Machine Learning Algorithms in Depth covers it

How deep it goes

Derives L2-norm margin optimization and regularized gradients, then implements both SVM and logistic-regression training.

Examples

Uses maximum-margin SVM geometry, an SVM implementation, and SGD logistic regression on a synthetic Gaussian-mixture dataset.

What it assumes

Assumes empirical loss, model parameters, classification constraints, gradients, and stochastic-gradient optimization.

What you can do after

Build regularized SVM and logistic-regression training procedures with explicit objectives, gradients, constraints, and parameters.

How it connects

Connects empirical risk, support vector machines, decision boundaries, quadratic programming, logistic regression, and stochastic gradient descent.

liveBookmanning.com

Read Machine Learning Algorithms in Depth on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningA regularization function is a function that assigns a penalty to model parameters or complexity as part of a regularized objective.
• 

  liveBookmanning.com
  Grokking Machine LearningRegularization is a model-training technique that penalizes complexity while optimizing predictive performance.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionRegularization is a modeling technique that constrains learning to reduce overfitting and improve generalization to unseen data.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIRegularization is a technique that limits model complexity by penalizing parameter values to reduce overfitting and improve generalization.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningThe practice of constraining a learned representation or model to encourage desirable structure such as smoothness or compactness.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA method that constrains a model during training to reduce overfitting and improve generalization.
• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackRegularization is a technique that constrains optimization to reduce undesirable or excessive fitting to an objective.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA set of techniques that reduce overfitting and improve generalization by constraining model behavior during training.
• 

  liveBookmanning.com
  Sutskever's ListA regularizer is a modeling mechanism that discourages undesirable solutions to improve generalization or control model complexity.
• 

  liveBookmanning.com
  Bayesian Optimization in ActionA loss function quantifies the discrepancy between a model's predictions and desired outcomes or data, with lower values indicating better fit.
• 

  liveBookmanning.com
  Causal AIRegularization is a modeling technique that adds a penalty or constraint to an optimization objective to discourage undesirable parameter values.
• 

  liveBookmanning.com
  Machine Learning System DesignRegularization is a technique that constrains model fitting to discourage overly complex solutions and improve generalization.

Compare how 13 books explain regularization →

Related concepts

• overfittingaddresses · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.
• dropoutincludes · Dropout is a neural-network regularization technique that randomly suppresses units, connections, or activations during training to reduce overfitting.
• regularization parameterconfigured by · A regularization parameter is a model-fitting setting that controls the influence of a penalty or constraint relative to fit to the data.
• variational autoencoderused by · A variational autoencoder is a generative neural network that learns a probabilistic latent representation of data and decodes samples from it to reconstruct or generate data.
• Kullback–Leibler divergenceuses · Kullback–Leibler divergence is a nonnegative, directed measure of the information lost when one probability distribution approximates another, computed as an expected log ratio under the first distribution.
• marginal likelihoodequivalent to · Marginal likelihood, or Bayesian evidence, is the probability or density of observed data under a model after marginalizing over unknown parameters or latent variables according to their prior distributions.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Algorithms in Depth. The section links point to the parts of the book it draws on.