liveBook · Manning

Slack variable

As explained in Timeless Algorithms, The Foundational Ideas

A nonnegative optimization variable measuring how much an observation violates a constraint, such as a classification margin.

A slack variable permits a controlled violation of an optimization constraint. In a soft-margin support vector machine (SVM), each training observation has a nonnegative slack variable ξᵢ that measures its violation of the classification margin. Slack makes it possible to preserve a wider margin without requiring every observation to satisfy the hard-margin constraint. section 8.3.2 section 8.3

For observation xᵢ with class label yᵢ, a candidate hyperplane has weight vector w and bias b. The hard-margin requirement is yᵢ(w·xᵢ + b) ≥ 1; the soft-margin constraint replaces its right side with 1 − ξᵢ, where ξᵢ ≥ 0. For that hyperplane, the smallest feasible slack is ξᵢ = max(0, 1 − yᵢ(w·xᵢ + b)). section 8.3.2 section 8.3.3

The minimum slack describes the observation’s position relative to the candidate boundary and margin:

  • ξᵢ = 0: the observation is on or beyond the correct margin boundary. section 8.3.2
  • 0 < ξᵢ < 1: the observation is inside the margin but on the correct side of the decision boundary. section 8.3.2
  • ξᵢ = 1: the observation lies on the decision boundary. section 8.3.2
  • ξᵢ > 1: the observation lies on the wrong side of the boundary and is misclassified. section 8.3.2

Slack variables are not features and do not move observations. Their values depend on the proposed hyperplane: changing w or b changes the boundary and margins, potentially changing every observation’s slack. section 8.3.2

The soft-margin objective minimizes ½||w||² + C∑ᵢξᵢ, subject to the relaxed margin constraints. The first term favors a wider margin; the second penalizes the total severity of violations. A large C makes slack expensive and encourages closer adherence to training observations. A small C tolerates more slack to preserve a wider margin and stronger regularization. section 8.3.3

Comparing candidate boundaries

For a wide-margin candidate, suppose two observations each have slack 0.30 and a misclassified observation has slack 1.20; total slack is 1.80. An illustrative moderate candidate reduces total slack to 1.35, while an aggressive candidate reduces it to 0.10 at the cost of a narrower margin. Among these three candidates, the wide boundary has the lowest objective at C = 0.10, the moderate boundary at C = 1, and the aggressive boundary at C = 100. These values illustrate the trade-off rather than estimates fitted from the plotted observations. section 8.3.4

Table 8.1 Comparison of the wide, moderately adjusted, and aggressive boundary candidates. Progressively larger weight-vector norms produce narrower margins but reduce total slack.
Candidate boundary
Weight-vector norm
Margin term
Margin width
Total slack
Wide
1.00
0.50
2.00
1.80
Moderately adjusted
1.30
0.85
1.54
1.35
Aggressive
2.19
2.40
0.91
0.10

The comparison separates each candidate’s margin width, margin term, and total slack. section 8.3.4

Allowing violations can reduce sensitivity to noise, outliers, or mislabeled observations, but it does not guarantee performance on unseen data. The result remains dependent on the sample, feature representation, penalty parameter, and any kernel used. section 10.2.7

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Derives relaxed margin constraints and minimum slack, then compares candidate boundaries through the soft-margin objective.
Examples
Compares wide, moderate, and aggressive boundaries using illustrative slack totals and three penalty values.
What you can do after
Evaluate candidate boundaries by interpreting slack values and their contribution to the soft-margin objective.
How it connects
Connects margin violations to regularization, generalization, and structural risk minimization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.