liveBook · Manning

Model diversity

As explained in Timeless Algorithms, The Foundational Ideas

Ensemble diversity is the extent to which constituent models differ in their structures, predictions, or errors.

Ensemble diversity is the extent to which constituent models differ in structure, predictions, or errors. Aggregation benefits when models make different enough errors to offset one another; adding more similar models yields diminishing benefit. section 9.2.3 section 9.3

Random forests create tree diversity through two forms of randomness:

  • Bootstrap sampling changes the observations used to train each tree. section 9.3
  • At each split, a randomly selected subset of features competes, so dominant predictors cannot shape every tree in the same way. section 9.3

Feature restrictions can make individual trees slightly less accurate, yet produce less-correlated errors and more effective aggregation. Diversity, rather than tree count alone, determines whether additional models contribute meaningfully. section 9.2.3 section 9.3

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Distinguishes tree count from diversity, then traces how bootstrap samples and random feature subsets produce less-correlated errors.
Examples
Illustrates correlated trees through dominant predictors repeatedly selected at splits, then shows random feature subsets as a remedy.
What you can do after
Evaluate ensemble growth by distinguishing additional tree count from added diversity and error compensation.
How it connects
Connects diversity to bagging, bootstrap sampling, random forests, feature selection, and prediction aggregation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.