The interpolation threshold is the model-capacity point at which a model first can fit every training example exactly. It marks a shift in the relationship between model size and test error: error can rise near the threshold, then fall as models become larger. section 8.5.2 section 8.5.3
A model just large enough to fit the training data may have a brittle solution: small amounts of noise or misspecification can damage its performance on new data. Beyond the threshold, many models can fit the training set, and stochastic gradient descent can find solutions that absorb noise while still generalizing. Extra capacity can therefore help a model find a simpler fit, rather than merely memorize more. section 8.5.3
Figure 8.9 Left: the classical bias–variance tradeoff. Training error (called “risk” in this image) is shown as a dashed line. It falls monotonically, whereas the test error (solid line) is U-shaped, with the bottom being optimal. Right: beyond the interpolation threshold, test error peaks and then declines again in the overparameterized (interpolating) regime, while training risk remains near zero. Used with permission granted by the lead author (Mikhail Belkin).
Compare the classical U-shaped test-error curve with the post-threshold decline in the overparameterized regime. section 8.5.2
A peak in model-size experiments
Across convolutional networks, ResNets, and Transformers, test error can dip, rise near the interpolation threshold, and dip again as models grow. A ResNet width curve illustrates the peak around the threshold. section 8.5.3
