Bias-variance tradeoff

As explained in Machine Learning Algorithms in Depth

The bias-variance tradeoff describes the balance between systematic prediction error from restrictive assumptions and error from sensitivity to training data.

The bias–variance tradeoff describes how mean squared error balances squared bias against variance. A biased estimator can achieve lower MSE when the variance reduction outweighs the added bias. section 7.8 section 7.3.1

Variance reduction connects the tradeoff to estimation limits and active data selection:

• The Cramér–Rao lower bound limits the variance of unbiased estimators when p(x|θ) satisfies the regularity condition. section 7.3.1
• The minimum-variance unbiased estimator reaches the inverse Fisher information matrix; multivariate models use a K × K information matrix. section 7.3.1
• An active learner can select data that maximizes Fisher information, using A-, D-, or E-optimality to reduce different summaries of inverse information. section 7.3.1
• Variance-reduction methods can be slower than uncertainty sampling because they invert a K × K matrix for every unlabeled instance, with complexity O(UK³). section 7.3.1

Active learning for logistic regression

A two-feature, two-class logistic-regression experiment compares random sampling with active-learning query strategies. The comparison includes uncertainty sampling and average KL divergence, with active learning shown to outperform random sampling in the associated evaluation. section 7.3.1

Figure 7.11 Uncertainty sampling techniques comparison. The left figure shows that active learning is better than random sampling. The right figure shows that average KL divergence is better than random sampling.

Compare uncertainty-sampling techniques and average KL divergence against random sampling in the plotted active-learning evaluation. section 7.3.1

How Machine Learning Algorithms in Depth covers it

How deep it goes

Derives the MSE relationship, connects variance to Fisher information and optimality criteria, then notes matrix-inversion costs.

Examples

Uses regression MSE, Fisher-information criteria, and a logistic-regression comparison of active-learning strategies with random sampling.

What you can do after

Apply it to choose variance-reduction strategies and evaluate active-learning queries against random sampling.

How it connects

Connects mean squared error, Cramér–Rao lower bound, Fisher information, active learning, uncertainty sampling, and query by committee.

liveBookmanning.com

Read Machine Learning Algorithms in Depth on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Sutskever's ListThe bias-variance tradeoff is the classical relationship in which predictive error reflects a balance between systematic approximation error and sensitivity to training-data variation.
• 

  liveBookmanning.com
  Machine Learning System DesignThe tradeoff between error from a model's restrictive assumptions and error from its sensitivity to variation in training data as model complexity changes.

Compare how 3 books explain bias-variance tradeoff →

Related concepts

• double descentcontrasts with · Double descent is a machine-learning performance pattern in which test error decreases, rises near an interpolation threshold, and then decreases again as model capacity or another resource increases.
• ensembleaddressed by · An ensemble is a technique that combines or considers outputs from multiple models or component methods collectively to produce or assess a joint result.
• overfittingcontrasts with · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.
• underfittingcontrasts with · A modeling failure in which insufficient capacity or training prevents a model from capturing relevant patterns in the data, resulting in poor performance even on training data.
• model complexitycontrasts with · Model complexity is the degree of structural or parametric flexibility and richness a model has for representing patterns and relationships.
• learning curveused by · A learning curve is a representation of how performance or proficiency changes as experience, training progress, or training data increases.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Algorithms in Depth. The section links point to the parts of the book it draws on.