liveBook · Manning

Generalization

As explained in Deep Learning with PyTorch, Second Edition

The ability of a trained model to perform well on data not seen during training by capturing underlying patterns rather than memorizing specifics.

A model generalizes when it learns properties that help it predict new samples, not just answers for the samples it saw during training. A face-to-age model, for example, should use signs of age rather than recognize particular people and recall their ages. section 14.5 section 14.5.1

Training and validation losses give a practical check:

  • If training loss does not fall, the model may be too simple, or the inputs may lack information needed for the prediction. section 5.5.3
  • If training loss falls but validation loss does not, the model is fitting its training samples without generalizing to unseen ones: overfitting. section 5.5.3
  • Similar downward trends are encouraging even when the two losses have different values. section 5.5.3
Figure 5.14 Overfitting scenarios when looking at the training (solid line) and validation (dotted line) losses. (A) Training and validation losses do not decrease; the model is not learning because there is no information in the data or the model’s capacity is insufficient. (B) Training loss decreases while validation loss increases: overfitting. (C) Training and validation losses decrease exactly in tandem. Performance may be improved further as the model is not at the limit of overfitting. (D) Training and validation losses have different absolute values but similar trends; overfitting is under control.
figure

Compare the solid training-loss and dotted validation-loss lines. Their trends reveal more than either loss alone. section 5.5.3

More parameters do not automatically mean better predictions on new data. A fully connected image model can memorize training images while struggling with subjects in different positions. A convolutional model uses locality and translation invariance; in a bird-versus-airplane comparison, it reached 90% validation accuracy with 18,090 parameters, versus 80% accuracy for a fully connected model with over 3.7 million. section 8.0 section 8.4.1

Choose augmentation that changes the useful example

With limited face images, flipping a photo left to right preserves its age-related properties while making it less trivial to memorize alongside the original. Randomly changing a few corner pixels can produce many variants, but the model can simply ignore those pixels. Effective augmentation encourages learning the class’s general properties, not incidental details. section 14.6

How Deep Learning with PyTorch, Second Edition covers it

How deep it goes
Explains generalization through loss trends, model capacity, architecture, and augmentation, without developing a formal mathematical account.
Examples
Uses thermometer fitting, face-to-age prediction, bird-versus-airplane classification, and mirrored face images.
What it assumes
Introduces the idea through everyday predictions while using PyTorch training and validation code for practical checks.
What you can do after
Diagnose/debug with it by comparing training and validation losses, then evaluate architecture and augmentation choices.
How it connects
Connects generalization to validation sets, overfitting, model capacity, convolutional neural networks, and data augmentation.
Cover of Deep Learning with PyTorch, Second Edition
liveBookmanning.com
Read Deep Learning with PyTorch, Second Edition on liveBook Get the book at manning.com

Also explained in

Compare how 13 books explain generalization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with PyTorch, Second Edition. The section links point to the parts of the book it draws on.