Variance measures how strongly a model’s error changes when the training data changes. A high-variance model is sensitive, less robust, and often overfits; a low-variance model is less sensitive but may underfit when paired with strong simplifying assumptions. section 1.3.2 section 1.4
Use variance to interpret the fit-versus-complexity tradeoff:
- Highly complex models usually have low bias and high variance: they fit training data closely but can generalize poorly. section 1.3.2
- Simpler models usually have high bias and low variance: they are less sensitive to noise but can underfit. section 1.3.2
- Ensembling reduces variance by averaging predictions, smoothing out individual estimator mistakes. section 1.4
- Diverse base estimators make averaging more useful because their errors and opinions differ, especially on difficult examples. section 1.4
Averaging six regressors
Train six different regression models on the Friedman-1 data set, collect each model’s test predictions, and average them for every sample. As ensemble size grows, the ensemble’s variance decreases, while its average performance generally improves. The standard deviation, defined here as the square root of variance, represents the width of the performance band.
