A falling training loss is good news—but only up to a point. Overfitting begins when a model keeps improving on its training data while its performance on unseen data deteriorates. It has learned patterns too specific to the training set rather than patterns that generalize. Watch validation metrics alongside training metrics; training performance alone cannot tell you when to stop. section 4.1.4 section 4.1.7
The fourth-epoch warning
In an IMDB sentiment classifier, training loss falls and training accuracy rises throughout a 20-epoch run. Validation performance peaks around epoch four, then worsens. More optimization is no longer helping the model generalize. One practical response is to build a fresh model, train it for four epochs, and evaluate it on the untouched test set; this run reaches about 88% accuracy. The exact best epoch can vary with random initialization. section 4.1.4
Listing 4.8 Plotting the training and validation loss
import matplotlib.pyplot as plt history_dict = history.history loss_values = history_dict["loss"] val_loss_values = history_dict["val_loss"] epochs = range(1, len(loss_values) + 1) plt.plot(epochs, loss_values, "r--", label="Training loss") #1 plt.plot(epochs, val_loss_values, "b", label="Validation loss") #2 plt.title("[IMDB] Training and validation loss") plt.xlabel("Epochs") plt.xticks(epochs) plt.ylabel("Loss") plt.legend() plt.show()
Plot training and validation loss from the Keras training history. A widening gap makes it easier to spot when additional epochs help the training set but not the validation set. section 4.1.4
Sometimes you should make a model overfit on purpose. First get a model with statistical power; then add layers, enlarge them, or train longer until validation performance starts to degrade. That establishes that the model has enough capacity to cross the boundary between underfitting and overfitting. The next job is to improve generalization, not to keep driving down training loss. section 6.2.4 section 5.5
Once validation performance turns, choose a response suited to the problem:
- Stop at a better epoch, or save the best validation-loss checkpoint so it can be reloaded later. section 4.1.4 section 8.2.4
- Reduce model capacity, add dropout or weight regularization, or improve the dataset. With little data, a smaller model can help. section 5.5 section 4.3.3
- For images, augment training samples with believable random changes such as flips, rotations, and zooms. The resulting inputs still come from the original images, so augmentation does not replace more independent data. section 8.2.5
- For recurrent models, consider recurrent dropout; its mask stays constant across timesteps rather than changing at every timestep. section 13.3.4
Overfitting can also happen to the validation process. Every architecture or hyperparameter choice made from validation results feeds information about that set into the final configuration. Repeated tuning can make validation performance look better than performance on genuinely new data, even without training directly on validation samples. Keep a separate test set for final evaluation. If its result is much worse, reconsider the validation procedure or use a more reliable protocol such as iterated K-fold validation. section 5.2.1 section 6.2.5
