Bias-variance tradeoff

As explained in Machine Learning System Design

The tradeoff between error from a model's restrictive assumptions and error from its sensitivity to variation in training data as model complexity changes.

The bias–variance tradeoff describes how model complexity changes generalization. More capacity can capture useful signals and reduce bias, but it can also increase sensitivity to training-data fluctuations and overfitting. section 9.1.1

Use the following distinctions when reading model performance:

• Underfitting reflects high bias: a model is too simple, misses important feature–target relationships, and performs poorly on both training and unseen data. section 9.1.1
• Overfitting reflects high variance: a model performs well on training data but learns insignificant patterns that fail to generalize. section 9.1.1
• Model complexity can involve tree depth, feature count, or neural-network layer count; increasing it usually lowers training error while increasing variance. section 9.1
• Learning-curve analysis helps assess whether changing complexity or adding training samples could improve performance. section 9.1

The U-shaped validation error

As model parameters increase, training error tends to keep falling while bias decreases. Variance rises, so validation error can form a U-shape; the practical goal is the balance near its minimum. section 9.1

Figure 9.2 With an increasing number of model parameters, training error tends to become lower and lower while minimizing bias. At the same time, the model variance increases, providing us with a U-shaped validation error.

Shows the opposing movement of training error, bias, variance, and validation error as model parameters increase. section 9.1

How Machine Learning System Design covers it

How deep it goes

Defines bias and variance, connects them to fitting behavior, and uses learning curves without deriving the tradeoff mathematically.

Examples

Uses tree depth, feature count, neural-network layers, training samples, and a U-shaped validation-error curve.

What you can do after

Diagnose underfitting or overfitting and evaluate complexity or additional-data choices with learning-curve analysis.

How it connects

Connects model complexity with learning curves, overfitting, underfitting, data leakage, and noisy features.

liveBookmanning.com

Read Machine Learning System Design on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Sutskever's ListThe bias-variance tradeoff is the classical relationship in which predictive error reflects a balance between systematic approximation error and sensitivity to training-data variation.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthThe bias-variance tradeoff describes the balance between systematic prediction error from restrictive assumptions and error from sensitivity to training data.

Compare how 3 books explain bias-variance tradeoff →

Related concepts

• double descentcontrasts with · Double descent is a machine-learning performance pattern in which test error decreases, rises near an interpolation threshold, and then decreases again as model capacity or another resource increases.
• ensembleaddressed by · An ensemble is a technique that combines or considers outputs from multiple models or component methods collectively to produce or assess a joint result.
• overfittingcontrasts with · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.
• underfittingcontrasts with · A modeling failure in which insufficient capacity or training prevents a model from capturing relevant patterns in the data, resulting in poor performance even on training data.
• model complexitycontrasts with · Model complexity is the degree of structural or parametric flexibility and richness a model has for representing patterns and relationships.
• learning curveused by · A learning curve is a representation of how performance or proficiency changes as experience, training progress, or training data increases.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.