A model overfits when it learns details specific to its training samples instead of patterns that help with new samples. Its training performance can keep improving while its performance on independent validation data gets worse. A flexible model has enough parameters to fit the points it sees, but a low training loss alone says little about what it will do between or beyond those points. section 5.5.3 section 14.5
Set aside a validation set before training. Fit the model on the remaining training set, then measure loss on both. Training loss tells you whether the model can fit the examples it sees; validation loss tests how it behaves on examples it did not train on. In a PyTorch training loop, compute both losses, but call backward() only on the training loss. section 5.5.3
Figure 5.14 Overfitting scenarios when looking at the training (solid line) and validation (dotted line) losses. (A) Training and validation losses do not decrease; the model is not learning because there is no information in the data or the model’s capacity is insufficient. (B) Training loss decreases while validation loss increases: overfitting. (C) Training and validation losses decrease exactly in tandem. Performance may be improved further as the model is not at the limit of overfitting. (D) Training and validation losses have different absolute values but similar trends; overfitting is under control.
Compare the solid training-loss and dotted validation-loss curves. Falling training loss alongside rising validation loss is the warning sign; different loss values with similar trends do not, by themselves, show overfitting. section 5.5.3
Remembering a face instead of learning age
Imagine predicting age from a face photo. A useful model learns age-related clues and applies them to a new person. An overfit model remembers identifying details and the ages of people it has seen. It may give a younger lookalike the older person’s age. This becomes more likely when model capacity is high relative to the available training samples. section 14.5.1
Check the metric that matters, not just an overall average. In a heavily imbalanced nodule classifier, positive-sample training loss fell nearly to zero while positive-sample validation loss rose. Overall loss looked fine because far more negative samples dominated it. At that point, continuing training was unlikely to improve positive-sample performance. section 14.4.3
To reduce the chance of memorization:
- Increase the number of training samples, or use meaningful data augmentation when new samples are unavailable. A mirrored face can retain age-related information while being less trivial to memorize than a few randomly changed corner pixels. section 8.5.1 section 14.6
- Consider capacity and architecture. Greater width or depth can help learn complex patterns but also make memorization easier. For images, convolutional layers address a limitation of fully connected classifiers that otherwise spend capacity learning translated replicas. section 7.2.6 section 8.8
- Use regularization where appropriate; weight decay, dropout, and batch normalization are applied in a convolutional-network example. section 8.8
