A model’s test error can follow a surprising path as its size grows: it decreases, rises near the point where the model first fits the training data, then decreases again. This second decline can outperform the classical bias–variance regime. section 8.5.3
Near the interpolation threshold, a model that barely fits the training data may have a brittle solution: noise or misspecified labels can damage its global structure and raise test error. A larger, overparameterized model has many ways to fit the data; stochastic gradient descent can find a solution that absorbs noise while still performing well on new data. section 8.5.3
The phenomenon was reported in 2018 and demonstrated across convolutional neural networks, ResNets, and Transformers in 2019. From a complexity perspective, a model barely large enough to fit the data may do so in a complex way, while a larger model can find a simpler explanation. section 8.5.3
Figure 8.10 Test (solid line) and training (dashed line) errors for ResNet as the width increases. A peak near the interpolation threshold (dashed line; shaded “critical” band) illustrates double descent, in which error rises as the model fits the data, then falls again for larger widths. Used with permission granted by the lead author (Preetum Nakkiran).
The ResNet plot shows test error peaking near the interpolation threshold and declining again as width increases. section 8.5.3
