Generalization is a model’s ability to perform effectively on new data from the same underlying task or distribution. A generalizing model learns useful patterns rather than memorizing training examples. section 8.13
Imagine studying for an exam. Memorizing practice-test answers may produce a perfect practice score but fail when questions are worded differently. Understanding the concepts supports adaptation to unseen problems; overfitting does not. section 8.13
Use these checks to distinguish learning from memorization:
- Reserve part of the data as a validation set that the model does not train on. section 11.34
- Compare training and validation loss during training. Falling training loss alongside rising validation loss signals that memorization is replacing useful learning. section 11.34
- Apply early stopping when validation loss stops improving; patience controls how many evaluation cycles to wait. section 11.34
- Improve weak results by choosing better features or a more suitable training algorithm. section 8.13
Language-model generalization
Larger training sets provide more tokens, while additional model weights provide capacity to learn fine relationships, rare words, edge cases, and varied writing styles. These capabilities support sensible answers to new prompts. section 11.3
Figure 8.15 A refresher on the machine learning life cycle
Use the machine learning life-cycle refresher to situate training, evaluation, and generalization within the broader workflow. section 8.13
