A slack variable permits a controlled violation of an optimization constraint. In a soft-margin support vector machine (SVM), each training observation has a nonnegative slack variable ξᵢ that measures its violation of the classification margin. Slack makes it possible to preserve a wider margin without requiring every observation to satisfy the hard-margin constraint. section 8.3.2 section 8.3
For observation xᵢ with class label yᵢ, a candidate hyperplane has weight vector w and bias b. The hard-margin requirement is yᵢ(w·xᵢ + b) ≥ 1; the soft-margin constraint replaces its right side with 1 − ξᵢ, where ξᵢ ≥ 0. For that hyperplane, the smallest feasible slack is ξᵢ = max(0, 1 − yᵢ(w·xᵢ + b)). section 8.3.2 section 8.3.3
The minimum slack describes the observation’s position relative to the candidate boundary and margin:
- ξᵢ = 0: the observation is on or beyond the correct margin boundary. section 8.3.2
- 0 < ξᵢ < 1: the observation is inside the margin but on the correct side of the decision boundary. section 8.3.2
- ξᵢ = 1: the observation lies on the decision boundary. section 8.3.2
- ξᵢ > 1: the observation lies on the wrong side of the boundary and is misclassified. section 8.3.2
Slack variables are not features and do not move observations. Their values depend on the proposed hyperplane: changing w or b changes the boundary and margins, potentially changing every observation’s slack. section 8.3.2
The soft-margin objective minimizes ½||w||² + C∑ᵢξᵢ, subject to the relaxed margin constraints. The first term favors a wider margin; the second penalizes the total severity of violations. A large C makes slack expensive and encourages closer adherence to training observations. A small C tolerates more slack to preserve a wider margin and stronger regularization. section 8.3.3
Comparing candidate boundaries
For a wide-margin candidate, suppose two observations each have slack 0.30 and a misclassified observation has slack 1.20; total slack is 1.80. An illustrative moderate candidate reduces total slack to 1.35, while an aggressive candidate reduces it to 0.10 at the cost of a narrower margin. Among these three candidates, the wide boundary has the lowest objective at C = 0.10, the moderate boundary at C = 1, and the aggressive boundary at C = 100. These values illustrate the trade-off rather than estimates fitted from the plotted observations. section 8.3.4
Table 8.1 Comparison of the wide, moderately adjusted, and aggressive boundary candidates. Progressively larger weight-vector norms produce narrower margins but reduce total slack.
|
Candidate boundary
|
Weight-vector norm
|
Margin term
|
Margin width
|
Total slack
|
|
Wide
|
1.00
|
0.50
|
2.00
|
1.80
|
|
Moderately adjusted
|
1.30
|
0.85
|
1.54
|
1.35
|
|
Aggressive
|
2.19
|
2.40
|
0.91
|
0.10
|
The comparison separates each candidate’s margin width, margin term, and total slack. section 8.3.4
Allowing violations can reduce sensitivity to noise, outliers, or mislabeled observations, but it does not guarantee performance on unseen data. The result remains dependent on the sample, feature representation, penalty parameter, and any kernel used. section 10.2.7
