liveBook · Manning

Regularization parameter

As explained in Timeless Algorithms, The Foundational Ideas

Regularization strength is a setting that controls how strongly a regularization penalty or constraint affects model fitting.

In a soft-margin support vector machine, the penalty parameter C sets the cost of margin violations relative to the preference for a wide margin. It controls how strongly the fitted boundary responds to the training observations. section 8.3.3 section 10.2.7

Objective= 12||w||2+C∑i=1nξi

The objective combines a term favoring a wider margin with a penalty on total slack. C scales that penalty. section 8.3.3

Changing C shifts the balance:

  • A larger C makes violations more costly, favoring closer fit to training observations and potentially a narrower margin. section 8.3.3
  • A smaller C tolerates more violations to preserve a wider margin; too small a value can lead to underfitting. section 8.3.3

C interacts with kernel parameters: for an RBF kernel, large C combined with large γ can produce a highly responsive boundary. Compare candidate settings with cross-validation rather than training accuracy alone. section 8.6.3 section 8.3.5

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines C within the soft-margin objective and explains its effects on margin, violations, and underfitting.
Examples
Uses a noisy or mislabeled observation to contrast hard-margin fitting with tolerated soft-margin violations.
What you can do after
Apply k-fold cross-validation to compare candidate C values using held-out classification measures.
How it connects
Connects support vector machines, slack variables, margin maximization, kernels, and cross-validation.

Also explained in

Compare how 4 books explain regularization parameter →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.