The fit-complexity tradeoff balances matching training data closely against keeping a model simple enough to generalize to future data. Underfitting reflects poor performance on both training and test data; overfitting reflects excellent training performance but poor test performance. section 1.3.2
The same tension is formally called the bias-variance tradeoff. Simpler models have higher bias but lower sensitivity to data noise, while complex models have lower bias but higher variance. Useful models sacrifice some fit and some simplicity to generalize effectively. section 1.3.2
Use the tradeoff to interpret model behavior:
- Increase complexity and training error may fall, but sensitivity to noise and overfitting risk rise. section 1.3.2
- Reduce complexity and the model may become flatter and more stable, but excessive simplicity can cause underfitting. section 1.3.2
- Combine models through ensemble methods to reduce variance among low-bias models or reduce bias among low-variance models. section 1.0 section 1.3.2
SVM complexity controlled by C
For an RBF-kernel support vector regressor, small values of C produce flatter, more linear models that can underfit. Large values produce curvier, more complex models with excellent training fit but greater overfitting risk. Validation curves compare training and test R² across C values to locate a useful balance. section 1.3.2
Figure 1.5 Support vector machine with an RBF kernel, with kernel parameter gamma = 0.75. Small values of C result in more linear (flatter) and less complex models that underfit the data, while large values of C result in more nonlinear (curvier) and more complex models that overfit the data. Selecting a good value for C is critically important in training a good SVM model.
Compare flatter and curvier RBF-kernel SVM regressors as C changes from low to high. section 1.3.2
