Generalization is a model’s ability to perform well on inputs it has never seen. Training loss measures something different: how well the model fits known examples. A model can keep improving on its training set while becoming worse on held-out data. That is overfitting. section 5.1 section 5.1.1
A model that learns but cannot generalize
Try replacing MNIST’s training labels with randomly shuffled labels: library(keras3) .[.[images, labels], .] <- dataset_mnist() images <- array_reshape(images / 255, c(60000, 28 * 28)) shuffled_labels <- sample(labels) model <- keras_model_sequential() |> layer_dense(512, activation = "relu") |> layer_dense(10, activation = "softmax") model |> compile(optimizer = "rmsprop", loss = "sparse_categorical_crossentropy", metrics = "accuracy") history <- model |> fit(images, shuffled_labels, epochs = 100, batch_size = 128, validation_split = 0.2) Training loss can fall because the network memorizes examples. Validation loss does not improve: shuffled labels provide no relationship to learn for new images. section 5.1.2
Figure 5.1 Canonical overfitting behavior
The overfitting curve shows the turning point: held-out performance improves early in training, then degrades even as training fit continues to improve. section 5.1.1
Useful data contains patterns that extend beyond individual samples. A deep network can learn to interpolate between examples along the data’s latent structure, but noisy labels, rare features, and irrelevant inputs can tempt it to fit misleading details instead. It needs enough representative data to learn the useful pattern, not just enough capacity to memorize. section 5.1.1 section 5.1.2
A practical workflow separates finding a learnable fit from improving its generalization:
- Establish a simple baseline and monitor training and validation metrics. If training improves but validation cannot beat the baseline, check whether the inputs contain useful information and whether the architecture suits the task. section 5.3 section 5.3.2
- Once the model shows useful validation performance, train enough to locate overfitting. A widening gap between training and validation performance tells you when further fitting is counterproductive. section 6.2.4 section 5.1.1
- Improve the data first where possible: collect more representative examples, correct labels, handle anomalies, and select or engineer informative features. section 5.4.1
- Then compare model sizes and try dropout, weight regularization, or early stopping. Select changes using validation data; reserve a separate test set for the final evaluation. section 5.4.4 section 6.2.5
For deep networks, this success is usually local: new inputs must remain close to patterns represented in training. Classifying a new pet photo under familiar variations is different from adapting to a wholly unanticipated task, which calls for broader or extreme generalization. section 19.2.2 section 20.2
