Generalization is a model’s ability to perform well on inputs it has never seen. A falling training loss is not proof of it: a sufficiently capable network can fit random labels while learning nothing useful for new inputs. The practical question is whether the model has learned a pattern that survives beyond its training examples. section 5.5 section 5.1.2
A model that fits but cannot generalize
Shuffle the labels on MNIST digits, then train a classifier. Its training loss falls, but validation performance does not improve: the labels contain no relationship the model could use on unseen digits. This is a quick reminder to check validation results, not just whether training runs successfully. section 5.1.2
Listing 5.4 Fitting an MNIST model with randomly shuffled labels
(train_images, train_labels), _ = mnist.load_data() train_images = train_images.reshape((60000, 28 * 28)) train_images = train_images.astype("float32") / 255 random_train_labels = train_labels[:] #1 np.random.shuffle(random_train_labels) model = keras.Sequential( [ layers.Dense(512, activation="relu"), layers.Dense(10, activation="softmax"), ] ) model.compile( optimizer="rmsprop", loss="sparse_categorical_crossentropy", metrics=["accuracy"], ) model.fit( train_images, random_train_labels, epochs=100, batch_size=128, validation_split=0.2, )
The shuffled-label Keras experiment makes the difference between fitting examples and generalizing to held-out data concrete. section 5.1.2
Why can a model generalize when the labels do contain a pattern? Real inputs occupy structured regions of the space of possible inputs. Handwritten digits, for example, represent only a tiny fraction of all possible pixel arrays; nearby valid examples can vary smoothly. A deep network can learn a useful approximation of this latent structure and interpolate between training samples. That also sets a limit: its predictions can stop making sense when inputs stray too far from what training covered. section 5.1.2 section 5.5
Use a staged workflow rather than applying regularization immediately:
- Establish a meaningful baseline. If training succeeds but validation cannot beat it, check whether the inputs contain predictive information and whether the architecture suits the task. section 5.3 section 5.3.2
- Give the model enough capacity to fit and eventually overfit. Watch training and validation loss; validation degradation marks the boundary you have crossed. section 5.3.3 section 6.2.4
- Then improve validation performance: curate or enlarge the dataset, tune capacity, or try dropout and weight regularization. For images, believable random transformations can expose the model to more variation. section 5.4.1 section 5.4.4 section 8.2.5
Measure generalization without letting the measurement become another training target. Use a validation set to choose models and reserve an untouched test set for final evaluation. Hold-out validation is usually sufficient with plenty of data; K-fold methods help when a small validation split would be unreliable. Repeated tuning can leak information from validation results into model choices, so a strong validation score alone is not a guarantee of test performance. section 5.2.1 section 6.2.2 section 6.2.5
There is a wider distinction beyond ordinary held-out performance. Deep networks chiefly achieve local generalization: they handle new inputs resembling the variations represented in training. Broad generalization would handle unanticipated situations within a domain; extreme generalization would adapt to novel situations with little or no new data. An image classifier recognizing unfamiliar pet photos is not, by that fact alone, equipped to solve an entirely new kind of problem. section 19.2.2 section 19.2.4
Testing adaptation rather than familiarity
An ARC-AGI task gives a few input–output grid examples and asks for the output to a new input. Its purpose is to probe adaptation to tasks the system was not trained or programmed to handle, rather than only performance on familiar variations of a fixed task. section 19.3.4
