Interpolation fills in gaps between observed examples. In deep learning, the useful gaps are often on a learned approximation of a data manifold: a structured region containing plausible inputs. A model can make sense of a nearby, unseen input by relating it to examples it has encountered. This is a source of local generalization, not a way to handle arbitrary novelty. section 5.1.2 section 5.5
Morphing handwritten digits
Imagine one handwritten digit gradually changing into another while each intermediate image remains a plausible digit. That path illustrates interpolation on the digit manifold. Simply averaging the two images’ pixels is different: the result usually is not a valid digit. Where the path goes matters as much as its endpoints. section 5.1.2
Figure 5.8 Difference between linear interpolation and interpolation on the latent manifold. Every point on the latent manifold of digits is a valid digit, but the average of two digits usually isn’t.
Compare a path through plausible digits with a straight-line pixel average that leaves the digit manifold. section 5.1.2
The same idea appears in image generation. Interpolating between text embeddings supplied to a text-to-image diffusion model can produce a smooth transition between output images. Here interpolation helps explore how changes in an input representation affect generated results. section 17.4
Two practical limits matter:
- Nearby examples need enough coverage. A dense sampling of the input–output space gives a model a better basis for interpolation; collecting more data and correcting labeling errors can improve its chances of generalizing. section 5.4.1
- A smooth guess is not an exact procedure. A Transformer trained on many examples of digit addition can still make mistakes because interpolating among examples does not reliably encode the step-by-step addition algorithm. Fundamentally discrete tasks such as list sorting are likewise poor fits for smooth interpolation. section 19.1.3 section 5.4.1
Use interpolation as an explanation for why a model may recognize new variations close to its training data—not as a promise that it has learned a general rule. Improving training fit alone can eventually hurt generalization; evaluate on separate data rather than assuming a good fit means the gaps have been filled correctly. section 5.1.2 section 5.5
