Ensemble learning combines predictions from multiple models to improve generalization. With unstable learners such as decision trees, variation among models can become useful: aggregation stabilizes predictions when the models do not all make the same mistakes. section 9.0
The benefit depends on diversity. Correlated errors limit what aggregation can correct: averaging similar predictions or voting among similarly mistaken trees may do little, even as more trees are added. section 9.2.3
Bagging and boosting create ensembles in different ways:
- Bagging trains models independently on bootstrap samples and combines predictions by averaging or voting; it primarily reduces variance. section 9.2.3
- Boosting trains models sequentially, directing each new model toward observations handled poorly by earlier models. section 9.2.3
- Random forests extend bagging by adding randomness to the features considered at each tree split. section 9.2.3















