Bagging, or bootstrap aggregation, reduces prediction variance by training multiple base learners on randomly selected bootstrap samples and combining their outputs. For regression, aggregate predictions by averaging; for classification, use voting. If the learners are f₁,…,fₘ, regression prediction can be written as f̂(x) = (1/M)∑ₘ fₘ(x). section 7.5.1
Construct a bagging ensemble as follows:
- Choose a base learning algorithm, such as a decision tree or K-nearest neighbors classifier. section 7.5.1
- Generate multiple training subsets by sampling observations randomly with replacement. section 7.5.1
- Train one base learner on each subset, typically in parallel. section 7.5 section 7.5.1
- Combine the resulting predictions with voting for classification or averaging for regression. section 7.5.1
- Increase the ensemble only while accuracy improves; beyond the observed plateau, additional estimators increase computational complexity without improving accuracy. section 7.5.1
Iris classification
On the iris dataset, bagging ensembles use 10 base estimators with 0.8 subsampling of training data and 0.8 subsampling of features. A bagged decision tree outperforms a bagged KNN ensemble because KNN is less sensitive to training-sample perturbations. Accuracy improves until roughly 20 estimators, then plateaus. section 7.5.1
