Overparameterization

As explained in Sutskever's List

An overparameterized model has more adjustable parameters or representational degrees of freedom than are required to fit its training data.

An overparameterized model has far more parameters than needed to fit its training data, so it can interpolate every example. This challenges the classical expectation that increasing model complexity eventually worsens test performance: beyond the interpolation threshold, test error can fall again. section 8.5.2

Near the threshold, a model that barely fits the data may have a brittle solution that is sensitive to noise. Larger models have many interpolating solutions; stochastic gradient descent can find one that fits the training data while generalizing well. Regularization and the optimizer’s implicit biases can also guide this choice. section 8.5.2 section 8.5.3

AlexNet memorizes random labels

A smaller AlexNet reached 99.82% training accuracy on CIFAR-10 with randomized labels, but only 9.86% test accuracy. On ImageNet with random labels, it reached 95.20% top-1 training accuracy. These results show that fitting the training set does not by itself guarantee generalization. section 8.5.2

Figure 8.9 Left: the classical bias–variance tradeoff. Training error (called “risk” in this image) is shown as a dashed line. It falls monotonically, whereas the test error (solid line) is U-shaped, with the bottom being optimal. Right: beyond the interpolation threshold, test error peaks and then declines again in the overparameterized (interpolating) regime, while training risk remains near zero. Used with permission granted by the lead author (Mikhail Belkin).

Compare the classical U-shaped test-error curve with the renewed decline beyond the interpolation threshold. section 8.5.2

Figure 8.10 Test (solid line) and training (dashed line) errors for ResNet as the width increases. A peak near the interpolation threshold (dashed line; shaded “critical” band) illustrates double descent, in which error rises as the model fits the data, then falls again for larger widths. Used with permission granted by the lead author (Preetum Nakkiran).

See test error rise near the interpolation threshold and fall again as ResNet width increases. section 8.5.3

How Sutskever's List covers it

How deep it goes

Defines the interpolation threshold and explains how excess capacity, regularization, and SGD relate to renewed declines in test error.

Examples

Uses AlexNet experiments with randomized CIFAR-10 and ImageNet labels and plots error against ResNet width.

What you can do after

Understand why fitting every training example can coexist with good test performance in larger models.

How it connects

Connects model capacity to the bias–variance tradeoff, interpolation, regularization, stochastic gradient descent, and generalization.

liveBookmanning.com

Read Sutskever's List on liveBook Get the book at manning.com

Related concepts

• double descentproduces · Double descent is a machine-learning performance pattern in which test error decreases, rises near an interpolation threshold, and then decreases again as model capacity or another resource increases.
• generalizationcontrasts with · Generalization is the extension of a pattern, model, solution, or conclusion beyond the specific examples or cases from which it was derived.
• stochastic gradient descentproduced by · Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly selected small batches.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.