A model shows local generalization when it makes useful predictions for new inputs that resemble its training examples. A dog-versus-cat model, for instance, may classify pet photos it has never seen, provided they are close enough to the photos it learned from. Unlike a lookup table, it need not have seen the exact input before. section 19.2.2
Deep nets achieve this by learning a pattern in the data and interpolating between examples along that pattern, or manifold. Think of handwritten digits gradually morphing into one another: nearby points can still look like valid digits. Simply averaging the pixels of two digit images, however, usually does not produce a valid digit. Interpolation works best when the model has learned the relevant shape of the data, not merely a way to blend raw inputs. section 5.1.2
Figure 5.8 Difference between linear interpolation and interpolation on the latent manifold. Every point on the latent manifold of digits is a valid digit, but the average of two digits usually isn’t.
The figure contrasts interpolation along the digit manifold with averaging pixels directly. section 5.1.2
The limit is distance from familiar examples. A pet classifier can handle a new camera angle or lighting condition when such variation is well represented in its training data. As inputs depart from what the model learned, its input-to-output mapping can quickly stop making sense. Data augmentation helps by exposing the model to more variations, but it does not give the model a way to improvise through situations with little or no relevant data. section 19.2.2
When nearby examples are not enough
To learn rocket launch parameters from trials alone, a deep net would need a dense sampling of possible inputs. A person can instead use an abstract physical model to derive a solution in one or a few trials. This contrasts local generalization with reasoning that can handle much greater novelty. section 19.2.2
