A model can handle an unfamiliar input by relating it to nearby training examples. This is interpolation: filling in gaps within a learned, structured space of data. It helps explain why deep learning can generalize to inputs it has not seen, but mainly when those inputs are close to what it has learned. section 5.1.2 section 5.5
Morphing digits is not averaging pixels
Handwritten digits can gradually morph into one another along a path of valid digits. Simply averaging the pixels of two digit images usually does not produce a valid digit. The useful path follows the learned latent manifold, not necessarily a straight line through pixel space. section 5.1.2
Figure 5.8 Difference between linear interpolation and interpolation on the latent manifold. Every point on the latent manifold of digits is a valid digit, but the average of two digits usually isn’t.
Compare a path through valid digits with the pixel average of two digits; they are different kinds of interpolation. section 5.1.2
An interpolative program is a program-like behavior a model learned from training examples. Scaling a model does not, by itself, give it the ability to synthesize a brand-new program at inference time for a substantially novel situation. Even familiar situations can expose brittle behavior, including sensitivity to phrasing or confounding features. Algorithmic reasoning is a particular difficulty because many desired programs cannot be expressed as continuous morphs of a data manifold. section 19.2
Rule of thumb: expect interpolation to help most near well-represented training data; do not mistake it for open-ended invention. The same interpolative character can be useful in image generation: interpolating between text embeddings supplied to a diffusion model can produce a smooth change between output images. section 5.1.2 section 17.4
