Expressive power is a model class’s flexibility to represent functions or fit data. It is often associated with capacity, complexity, and the number of parameters. section 9.1.6 section 9.1.3
Use capacity and learning-curve behavior to decide whether a model is too simple, too flexible, or operating in a less predictable regime.
- Low capacity can cause underfitting: the model misses important relationships and performs poorly on both training and unseen data. section 9.1.1
- High capacity can reduce bias by capturing useful signals, but it can increase variance and produce overfitting. section 9.1.1
- When training loss falls while validation loss rises, restrict capacity by reducing complexity or increasing regularization. section 9.1.3
- Do not treat the classical bias–variance pattern as universal: highly overparameterized networks can exhibit double descent as capacity increases. section 9.1.6
A practical capacity check
Suppose training performance keeps improving while validation performance worsens. Treat the pattern as possible high variance, then reduce model complexity or increase regularization rather than assuming more capacity will help. section 9.1.3 section 9.1.1
Figure 9.6 Loss is decreasing for training but not validation, reflecting potential overfit.
The loss-curve figure makes the training-versus-validation signature of potential overfitting immediately visible. section 9.1.3
