Imagine studying for an exam by memorizing every answer instead of learning how to solve new questions. You might ace the questions you have seen and struggle with unfamiliar ones. Overfitting is the modeling version: a model fits its training data extremely well but performs poorly on new data. It often happens when the model is too complex for the pattern it needs to learn. A model that is too simple has the opposite problem, underfitting. section 4.0 section 4.7
A curve that fits too well
Suppose a small dataset looks roughly like a downward-opening parabola, with a little noise. A straight line misses the pattern. A degree-2 polynomial captures it. A degree-10 polynomial can pass through every training point, but its extra bends follow the noise rather than the underlying shape. Perfectly fitting the points you have is not necessarily the goal; predicting points you have not seen is. section 4.1
Compare errors to spot the difference. An underfit model has high training and testing error; a useful fit has low error on both. An overfit model has very low training error but high testing error. If increasing complexity keeps lowering training error while error on held-out data rises, more complexity is hurting rather than helping. section 4.2 section 4.4
Figure 4.5 The model complexity graph is an effective tool to help us determine the ideal complexity of a model to avoid underfitting and overfitting. In this model complexity graph, the horizontal axis represents the degree of several polynomial regression models, from 0 to 10 (i.e., the complexity of the model). The vertical axis represents the error, which in this case is given by the mean absolute error. Notice that the training error starts large and decreases as we move to the right. This is because the more complex our model is, the better it can fit the training data. The validation error, however, starts large, then decreases, and then increases again—very simple models can’t fit our data well (they underfit), whereas very complex models fit our training data but not our validation data because they overfit. A happy point in the middle is where our model neither underfits or overfits, and we can find it using the model complexity graph.
Follow both curves as polynomial degree increases: training error falls, but validation error eventually rises. The useful region lies before that rise. section 4.4
How can you avoid choosing a model that only looks good on familiar data?
- Separate the jobs of your data: train models on the training set, choose a model or its hyperparameters using the validation set, and use the testing set only for the final check. Choosing based on test results would make the test part of your decision process. section 4.3
- Compare training and validation error across model complexities. A complexity graph helps locate the point where validation error is low; a simpler model with only slightly higher validation error may also be worth choosing. section 4.4
- Use regularization when training: add a complexity term to the error being minimized, weighted by a parameter λ. L1 regularization can drive coefficients to zero; L2 tends to shrink them without making them zero. section 4.5 section 4.7
- For a decision tree, limit excessive splitting with conditions such as maximum depth or minimum samples per leaf. A neural network example uses dropout layers to help prevent overfitting. section 9.2 section 10.3
Try diagnosing four models
Given training and testing errors of (0.1, 1.8), (0.4, 1.2), (0.6, 0.8), and (1.9, 2.3), which model looks overfit, which looks underfit, and which has the lowest testing error? The first has the starkest training-to-testing gap; the fourth performs poorly on both; the third has the lowest testing error. For an actual model-selection workflow, make that choice using validation data and reserve testing data for the final check. section 4.8 section 4.3
