chapter nine

9 From single trees to random forests: Leo Breiman and the logic of ensemble learning

This chapter covers

Leo Breiman’s Random Forests (2001) and the emergence of a powerful new approach to ensemble learning
Why flexible, interpretable decision trees often fail to generalize reliably
How bootstrap sampling and aggregation stabilize predictions by reducing variance
How random feature selection creates the diversity that turns bagged trees into random forests
How strength, correlation, out-of-bag evaluation, and variable importance explain and reveal the forest’s behavior

The previous chapter examined how Vladimir Vapnik confronted one of machine learning’s central problems: models that perform well on training data but fail to generalize to new observations. His response was to develop a fundamentally new classification method. Support vector machines pursued generalization by constructing a carefully controlled decision boundary, balancing separation, complexity, and classification error.

9.1 The strengths and limitations of single decision trees

9.1.1 Instability and sensitivity to early splits

9.1.2 Depth, overfitting, and variance

9.1.3 Why pruning is not enough

9.2 Bagging: randomness in the observations

9.2.1 How bagging creates different training sets

9.2.2 Aggregation: turning variation into stability

9.2.3 What bagging fixes—and what it does not

9.3 Random forests: randomness in the features

9.3.1 Random feature selection at each split

9.3.2 Creating diversity by limiting choice

9.3.3 The complete random forest algorithm

9.4 Why random forests generalize

9.4.1 Classification margins: measuring collective confidence

9.4.2 Balancing tree strength and correlation

9.4.3 Why adding trees stabilizes the forest

9.5 Out-of-bag evaluation

9.6 Variable importance: recovering interpretation

9.6.1 Measuring variable importance

9.6.2 interpreting importance responsibly

9.7 Random forests versus support vector machines

9.8 The enduring legacy of random forests

9.9 Summary