A single decision tree can change substantially when its training data changes. Bagging, or bootstrap aggregation, addresses this instability by training models on repeated samples of the data and combining their predictions. Rather than simplifying each tree, it seeks stability in their collective result. Leo Breiman introduced the general method in 1996 for unstable learners, including trees, neural networks, and procedures involving variable selection. section 9.1.3 section 9.2
For a dataset of N observations, a bootstrap sample is formed by drawing N times with replacement. An observation can therefore appear more than once, while another may be omitted. Bagging uses bootstrap sampling to create varied training sets, fits a model to each, and then aggregates their predictions; sampling alone is not bagging. section 9.2.1 section 9.9
Figure 9.1 Bagging with decision trees. Bootstrap sampling creates multiple training sets by drawing observations with replacement, so each sample contains duplicated and omitted observations. A separate decision tree is trained on each sample using the same procedure. The trees’ predictions are then combined by majority vote for classification or averaging for regression, producing a more stable final prediction.

The diagram traces bagging from bootstrap samples through separate trees to a combined prediction. section 9.2.2
For classification, the ensemble selects the class with the most votes. For regression, it averages numerical predictions: ŷ(x) = (1/B) ∑ T_b(x), where B is the number of trees and T_b(x) is tree b’s prediction for observation x. Averaging can balance unusually high predictions against lower ones; voting can overrule an individual tree’s error. section 9.2.2 section 9.2.3
A classification vote
If 62 of 100 trees predict one class and 38 predict another, the ensemble chooses the class supported by 62 trees. section 9.2.2
The individual trees remain sensitive to their samples. Aggregation reduces variance at the ensemble level when sample-specific effects differ across trees: patterns appearing in only a few trees have less influence than patterns that recur. Unlike pruning, bagging can retain deep trees’ ability to capture nonlinear relationships and interactions. section 9.2.2
The benefit depends on diversity. A strong predictor may be selected near the root of many trees, producing similar structures and correlated errors. Voting or averaging cannot reliably cancel a mistake shared by most trees. Random forests extend bagging by also randomizing the features considered at each split, making varied tree structures more likely. section 9.2.3 section 9.9





