A model can fit its training data well yet produce a different prediction when trained on another sample from the same population. Model variance is this sensitivity to the training sample. High variance can make strong training performance a poor guide to performance on new observations. section 9.1.2
Deep decision trees illustrate the problem. A small change in the observations can change an early split, reshape later branches, and alter terminal predictions. Deep trees can capture complex relationships, but their localized partitions may also fit sample-specific noise. Restricting or pruning a tree can improve stability, at the cost of limiting what it can capture. section 9.1.2 section 9.2.3
Bagging takes a different approach: train trees on bootstrap samples and aggregate their predictions. For regression, the combined prediction is ŷbag(x) = (1/B) ∑ᵦ Tᵦ(x), where B is the number of trees and Tᵦ(x) is tree b’s prediction for x. Averaging can dilute effects peculiar to individual samples without making each tree stable. For classification, trees instead vote on the predicted class. section 9.2.2
Figure 9.1 Bagging with decision trees. Bootstrap sampling creates multiple training sets by drawing observations with replacement, so each sample contains duplicated and omitted observations. A separate decision tree is trained on each sample using the same procedure. The trees’ predictions are then combined by majority vote for classification or averaging for regression, producing a more stable final prediction.

The diagram traces bootstrap samples through separate trees to an aggregated prediction. section 9.2.2
Aggregation is limited when trees make correlated errors: a mistake shared by most trees cannot be canceled by their votes or averages. Random forests encourage diversity by also restricting each split to a randomly selected subset of features. The ensemble can reduce instability, but it cannot eliminate uncertainty about unseen observations. section 9.2.3 section 10.2.8
Variance can also be managed within a single model. In a soft-margin support vector machine, a very large penalty parameter C can make the boundary overly responsive to individual observations; a very small C can make it too rigid. These are tendencies, not guarantees, and no C is universally best. section 8.3.3
Model variance should be distinguished from error variance. In normal-error regression, error variance describes the spread of the error distribution around the fitted line, rather than how the fitted model changes across training samples. section 3.4.3 section 9.1.2























