Regularization limits how strongly a model adapts to training observations, reducing reliance on noise or accidental relationships that may not persist after deployment. It need not remove layers, nodes, or parameters; it can instead constrain how the model uses its capacity. section 11.6.2
Two examples illustrate how regularization shapes learning:
- Weight decay penalizes large neural-network weights, discouraging reliance on a small number of large weights. section 11.6.2
- In a soft-margin support vector machine, the penalty parameter C sets the cost of slack: large C favors fewer violations and a more responsive boundary; small C tolerates more violations and favors a wider margin. section 8.3.3
The SVM objective balances a small squared weight norm, associated with a wider margin, against the penalty C times total slack. section 8.3.3
The choice involves a tradeoff, not a universally best setting. Excessively strong regularization can make a model too rigid and underfit; too little can make it overly responsive to training observations and overfit. These are tendencies, not guarantees. section 8.3.3
















