Regularization parameter

As explained in Ensemble Methods for Machine Learning

A regularization parameter controls the strength of the complexity penalty relative to the data-fitting loss.

A regularization parameter sets the balance between fitting the training data and keeping the model simple. Stronger regularization favors flatter, less complex models; weaker regularization permits greater complexity. section 1.3.2 section 6.2.2

Read the parameter direction carefully:

• For support vector regression, a small C emphasizes simplicity, increasing training error and risking underfitting. section 1.3.2
• A large C emphasizes training fit, allowing curvier models with lower training error and greater overfitting risk. section 1.3.2
• With alpha, the direction reverses: large alpha favors regularization, while small alpha favors training fit; C is essentially the inverse of alpha. section 6.2.2

Choosing C for an SVR

Train SVR models across a logarithmic range of C values, compare training and test R2 scores with validation_curve, and prefer the value that generalizes rather than merely fitting the training data. On Friedman-1 data, increasing C produces increasingly complex models whose training fit can improve while generalization deteriorates. section 1.3.2

Figure 1.5 Support vector machine with an RBF kernel, with kernel parameter gamma = 0.75. Small values of C result in more linear (flatter) and less complex models that underfit the data, while large values of C result in more nonlinear (curvier) and more complex models that overfit the data. Selecting a good value for C is critically important in training a good SVM model.

Compare how increasing C changes an RBF support vector regressor from flatter, potentially underfit behavior to curvier, potentially overfit behavior. section 1.3.2

How Ensemble Methods for Machine Learning covers it

How deep it goes

Explains the fit-complexity tradeoff, parameter-direction effects, and validation-based tuning, but does not derive optimization procedures.

Examples

Uses SVR models, RBF kernels, Friedman-1 regression data, validation curves, and regularized logistic-loss formulas.

What it assumes

Assumes familiarity with training loss and model fitting while establishing regularization as a fit-versus-complexity tradeoff.

What you can do after

Apply it via scikit-learn validation curves to compare complexity, training fit, and generalization across parameter values.

How it connects

Connects regularization parameters with loss functions, model complexity, support vector machines, support vector regression, underfitting, and overfitting.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningA training hyperparameter that controls the strength of the complexity penalty relative to predictive error.

Compare how 2 books explain regularization parameter →

Related concepts

• regularizationconfigures · Regularization is a modeling technique that constrains or penalizes solutions during fitting or optimization to discourage undesirable complexity or behavior and improve generalization or estimation.
• hyperparameterkind of · A hyperparameter is a setting chosen outside ordinary parameter learning that controls a model, algorithm, or learning procedure.
• cross-validationused by · Cross-validation is a resampling technique that evaluates a model by repeatedly fitting it on some portions of available data and assessing it on held-out portions.
• margincontrasts with · A classification margin measures separation or confidence, expressed as distance from a decision boundary or as the correct class's score advantage over competing classes.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.