A model’s expressive power is its flexibility to fit patterns in data. More complex models can fit training data more closely, but may overfit; simpler models may underfit. Complexity can be represented by polynomial degree or measured through coefficient norms. section 4.4 section 4.5
To compare polynomial models, plot training and validation errors against degree. Choose a degree with low validation error: very simple models can underfit, while very complex ones can overfit. section 4.4
Figure 4.5 The model complexity graph is an effective tool to help us determine the ideal complexity of a model to avoid underfitting and overfitting. In this model complexity graph, the horizontal axis represents the degree of several polynomial regression models, from 0 to 10 (i.e., the complexity of the model). The vertical axis represents the error, which in this case is given by the mean absolute error. Notice that the training error starts large and decreases as we move to the right. This is because the more complex our model is, the better it can fit the training data. The validation error, however, starts large, then decreases, and then increases again—very simple models can’t fit our data well (they underfit), whereas very complex models fit our training data but not our validation data because they overfit. A happy point in the middle is where our model neither underfits or overfits, and we can find it using the model complexity graph.
See how training error falls as polynomial degree increases while validation error eventually rises. section 4.4
