liveBook · Manning

Bagging

As explained in Timeless Algorithms, The Foundational Ideas

Bagging, or bootstrap aggregation, is an ensemble technique that trains models on bootstrap samples and combines their predictions to reduce variability.

A single decision tree can change substantially when its training data changes. Bagging, or bootstrap aggregation, addresses this instability by training models on repeated samples of the data and combining their predictions. Rather than simplifying each tree, it seeks stability in their collective result. Leo Breiman introduced the general method in 1996 for unstable learners, including trees, neural networks, and procedures involving variable selection. section 9.1.3 section 9.2

For a dataset of N observations, a bootstrap sample is formed by drawing N times with replacement. An observation can therefore appear more than once, while another may be omitted. Bagging uses bootstrap sampling to create varied training sets, fits a model to each, and then aggregates their predictions; sampling alone is not bagging. section 9.2.1 section 9.9

Figure 9.1 Bagging with decision trees. Bootstrap sampling creates multiple training sets by drawing observations with replacement, so each sample contains duplicated and omitted observations. A separate decision tree is trained on each sample using the same procedure. The trees’ predictions are then combined by majority vote for classification or averaging for regression, producing a more stable final prediction.

The diagram traces bagging from bootstrap samples through separate trees to a combined prediction. section 9.2.2

For classification, the ensemble selects the class with the most votes. For regression, it averages numerical predictions: ŷ(x) = (1/B) ∑ T_b(x), where B is the number of trees and T_b(x) is tree b’s prediction for observation x. Averaging can balance unusually high predictions against lower ones; voting can overrule an individual tree’s error. section 9.2.2 section 9.2.3

A classification vote

If 62 of 100 trees predict one class and 38 predict another, the ensemble chooses the class supported by 62 trees. section 9.2.2

The individual trees remain sensitive to their samples. Aggregation reduces variance at the ensemble level when sample-specific effects differ across trees: patterns appearing in only a few trees have less influence than patterns that recur. Unlike pruning, bagging can retain deep trees’ ability to capture nonlinear relationships and interactions. section 9.2.2

The benefit depends on diversity. A strong predictor may be selected near the root of many trees, producing similar structures and correlated errors. Voting or averaging cannot reliably cancel a mistake shared by most trees. Random forests extend bagging by also randomizing the features considered at each split, making varied tree structures more likely. section 9.2.3 section 9.9

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Explains bootstrap sampling and aggregation, their effect on variance, and the limit imposed by correlated errors.
Examples
Illustrates classification with 62 of 100 trees voting for one class and 38 for another.
What you can do after
Evaluate and understand how sampling and aggregation stabilize predictions and why shared errors limit the benefit.
How it connects
Connects bagging to decision trees, pruning, bootstrap sampling, and random forests.

Also explained in

Compare how 6 books explain bagging →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.