Bias-variance tradeoff

As explained in Sutskever's List

The bias-variance tradeoff is the classical relationship in which predictive error reflects a balance between systematic approximation error and sensitivity to training-data variation.

The classical picture links model capacity to predictive error: training error falls as models grow, while test error forms a U-shaped curve. Polynomial fitting makes the tradeoff concrete: a line underfits parabolic data, a quadratic fits it, and higher-degree polynomials can chase noise. section 8.5.2

That picture shaped early deep-learning practice. AlexNet’s 60 million parameters raised overfitting concerns, addressed with data augmentation, dropout, and weight decay. Yet larger deep models later generalized well even with modest or no explicit regularization, challenging the idea that they simply sat on the overfitting side of the U. section 8.5.2

Figure 8.9 Left: the classical bias–variance tradeoff. Training error (called “risk” in this image) is shown as a dashed line. It falls monotonically, whereas the test error (solid line) is U-shaped, with the bottom being optimal. Right: beyond the interpolation threshold, test error peaks and then declines again in the overparameterized (interpolating) regime, while training risk remains near zero. Used with permission granted by the lead author (Mikhail Belkin).

Compare the classical U-shaped test-error curve with the later decline beyond the interpolation threshold. section 8.5.2

In overparameterized models, capacity exceeds what is needed to fit the training data exactly; the interpolation threshold marks where exact fitting becomes possible. Beyond it, test error can rise near the threshold and then fall again—a pattern named double descent by Mikhail Belkin and colleagues in 2018. section 8.5.2 section 8.5.3

A model barely large enough to fit the data may have a brittle solution, highly sensitive to noise or misspecified labels. Larger models have many interpolating solutions; stochastic gradient descent can find one that fits the data while still generalizing. More capacity, then, need not mean a more complex explanation. section 8.5.3

Figure 8.10 Test (solid line) and training (dashed line) errors for ResNet as the width increases. A peak near the interpolation threshold (dashed line; shaded “critical” band) illustrates double descent, in which error rises as the model fits the data, then falls again for larger widths. Used with permission granted by the lead author (Preetum Nakkiran).

The ResNet plot shows error peaking near the interpolation threshold before declining as width increases. section 8.5.3

How Sutskever's List covers it

How deep it goes

Explains the classical error curve and gives an informal mechanism for double descent through solution sensitivity and SGD.

Examples

Uses polynomial fitting, AlexNet, and ResNet error curves to illustrate underfitting, regularization, and double descent.

What it assumes

Assumes familiarity with training and test error while defining overparameterization and the interpolation threshold.

What you can do after

Evaluate model-size behavior by distinguishing the classical U-shaped error curve from post-interpolation double descent.

How it connects

Connects model capacity to regularization, overparameterization, interpolation, and stochastic gradient descent.

liveBookmanning.com

Read Sutskever's List on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthThe bias-variance tradeoff describes the balance between systematic prediction error from restrictive assumptions and error from sensitivity to training data.
• 

  liveBookmanning.com
  Machine Learning System DesignThe tradeoff between error from a model's restrictive assumptions and error from its sensitivity to variation in training data as model complexity changes.

Compare how 3 books explain bias-variance tradeoff →

Related concepts

• double descentcontrasts with · Double descent is a machine-learning performance pattern in which test error decreases, rises near an interpolation threshold, and then decreases again as model capacity or another resource increases.
• ensembleaddressed by · An ensemble is a technique that combines or considers outputs from multiple models or component methods collectively to produce or assess a joint result.
• overfittingcontrasts with · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.
• underfittingcontrasts with · A modeling failure in which insufficient capacity or training prevents a model from capturing relevant patterns in the data, resulting in poor performance even on training data.
• model complexitycontrasts with · Model complexity is the degree of structural or parametric flexibility and richness a model has for representing patterns and relationships.
• learning curveused by · A learning curve is a representation of how performance or proficiency changes as experience, training progress, or training data increases.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.