liveBook · Manning

Ensemble

As explained in Timeless Algorithms, The Foundational Ideas

Ensemble learning combines multiple predictive models to improve predictions.

Ensemble learning combines predictions from multiple models to improve generalization. With unstable learners such as decision trees, variation among models can become useful: aggregation stabilizes predictions when the models do not all make the same mistakes. section 9.0

The benefit depends on diversity. Correlated errors limit what aggregation can correct: averaging similar predictions or voting among similarly mistaken trees may do little, even as more trees are added. section 9.2.3

Bagging and boosting create ensembles in different ways:

  • Bagging trains models independently on bootstrap samples and combines predictions by averaging or voting; it primarily reduces variance. section 9.2.3
  • Boosting trains models sequentially, directing each new model toward observations handled poorly by earlier models. section 9.2.3
  • Random forests extend bagging by adding randomness to the features considered at each tree split. section 9.2.3

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Explains aggregation, variance reduction, model diversity, and how correlated errors limit ensemble gains.
Examples
Uses bagged trees, random forests, AdaBoost, gradient boosting, and XGBoost as examples of ensemble methods.
What you can do after
Evaluate ensemble behavior by distinguishing variance reduction from the limits imposed by shared errors.
How it connects
Links ensemble learning to decision trees, bagging, boosting, random forests, bootstrap sampling, variance, and correlated errors.

Also explained in

Compare how 16 books explain ensemble →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.