liveBook · Manning

Variance

As explained in Timeless Algorithms, The Foundational Ideas

Variance is the model's sensitivity to changes in the training sample, reflected in how its behavior varies across samples.

A model can fit its training data well yet produce a different prediction when trained on another sample from the same population. Model variance is this sensitivity to the training sample. High variance can make strong training performance a poor guide to performance on new observations. section 9.1.2

Deep decision trees illustrate the problem. A small change in the observations can change an early split, reshape later branches, and alter terminal predictions. Deep trees can capture complex relationships, but their localized partitions may also fit sample-specific noise. Restricting or pruning a tree can improve stability, at the cost of limiting what it can capture. section 9.1.2 section 9.2.3

Bagging takes a different approach: train trees on bootstrap samples and aggregate their predictions. For regression, the combined prediction is ŷbag(x) = (1/B) ∑ᵦ Tᵦ(x), where B is the number of trees and Tᵦ(x) is tree b’s prediction for x. Averaging can dilute effects peculiar to individual samples without making each tree stable. For classification, trees instead vote on the predicted class. section 9.2.2

Figure 9.1 Bagging with decision trees. Bootstrap sampling creates multiple training sets by drawing observations with replacement, so each sample contains duplicated and omitted observations. A separate decision tree is trained on each sample using the same procedure. The trees’ predictions are then combined by majority vote for classification or averaging for regression, producing a more stable final prediction.

The diagram traces bootstrap samples through separate trees to an aggregated prediction. section 9.2.2

Aggregation is limited when trees make correlated errors: a mistake shared by most trees cannot be canceled by their votes or averages. Random forests encourage diversity by also restricting each split to a randomly selected subset of features. The ensemble can reduce instability, but it cannot eliminate uncertainty about unseen observations. section 9.2.3 section 10.2.8

Variance can also be managed within a single model. In a soft-margin support vector machine, a very large penalty parameter C can make the boundary overly responsive to individual observations; a very small C can make it too rigid. These are tendencies, not guarantees, and no C is universally best. section 8.3.3

Model variance should be distinguished from error variance. In normal-error regression, error variance describes the spread of the error distribution around the fitted line, rather than how the fitted model changes across training samples. section 3.4.3 section 9.1.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines sample sensitivity and explains tree instability, aggregation, and correlated-error limits without deriving a formal variance decomposition.
Examples
Uses deep trees with changing splits and soft-margin SVMs with different penalty strengths to illustrate sample sensitivity.
What you can do after
Enables evaluation of how tree constraints, aggregation, and SVM regularization address sensitivity to training samples.
How it connects
Links variance to bias, overfitting, pruning, bootstrap sampling, bagging, random forests, and support vector machine regularization.

Also explained in

Compare how 24 books explain variance →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.