Model complexity is the structural or parametric richness that controls how broad a range of functions a model can represent. Increasing complexity usually improves fit to training data, but can damage performance on unseen data through overfitting. section 9.1
Treat complexity as a model-selection variable rather than an automatic improvement:
- Choose a hyperparameter that represents complexity, such as tree depth, feature count, regularization strength, or neural-network layer count. section 9.1.4
- Define a grid of candidate values, train each model to convergence, and record its final loss or evaluation metric. section 9.1.4
- Plot performance against the chosen complexity measure to identify a useful range for the available data. section 9.1.4
- Reduce complexity or gather more data when variance is high; use a more complicated model when persistent high error indicates bias. section 9.1.5
Choosing a model size
Suppose tree depth is the complexity control. Train models at depths 2 through 16, capture their converged metrics, and inspect the resulting curve. Select the range that balances training fit with validation performance instead of automatically choosing the deepest tree. section 9.1.4
Figure 9.2 With an increasing number of model parameters, training error tends to become lower and lower while minimizing bias. At the same time, the model variance increases, providing us with a U-shaped validation error.
Use the bias-variance diagram to see why training error can keep falling while validation error eventually rises as parameter count increases. section 9.1
Figure 9.7 Finding the optimal model complexity based on learning curves
Use the model-wise learning-curve figure to compare candidate complexity settings and locate an effective operating range. section 9.1.4
