chapter nine
9 From single trees to forests: Leo Breiman and the logic of ensemble learning
This chapter covers
- Leo Breiman’s Random Forests (2001) and the emergence of ensemble learning from unstable single trees
- Why decision trees, despite their flexibility, fail to generalize due to instability and high variance
- How bootstrap sampling, feature randomness, and aggregation combine to stabilize predictions and improve generalization on new data
- How Breiman’s strength-correlation framework explains why ensembles succeed where single models fail
- How random forests balance predictive power with interpretability through internal diagnostics such as out-of-bag error and variable importance
The previous chapter examined how Vladimir Vapnik confronted one of the earliest and most persistent failures of machine learning: models that achieved impressive accuracy on training data yet performed unreliably on new examples. Vapnik’s response was to redefine learning itself. Rather than chasing accuracy alone, support vector machines—particularly soft-margin SVMs—sought generalization by explicitly controlling model capacity, balancing margin width against classification error, and grounding learning in statistical theory. Geometry became a disciplined safeguard against overfitting rather than a mere visualization of decision boundaries.