This chapter covers:
- What are ensemble methods?
- What are bagging, boosting, and stacking, and why are they useful?
- Using the random forest and XGBoost algorithms to predict animal classes
- How to benchmark multiple algorithms against the same task
In the last chapter I showed you how we can use the recursive partitioning algorithm to train decision trees that are very interpretable. We finished by highlighting an important limitation of decision trees: they have a tendency to overfit the training set. This results in models that generalize poorly to new data. As a result, individual decision trees are rarely used, but can become extremely powerful predictors when many trees are combined together.
By the end of this chapter you’ll understand the difference between ordinary decision trees and ensemble methods, such as random forest and gradient boosting, which combine multiple trees to make predictions. Finally, as this is the last chapter in the classification part of the book, you’ll learn what benchmarking is, and how to use it to find the best performing algorithm for a particular problem. Benchmarking is the process of letting a bunch of different learning algorithms fight it out to select the one that performs the best for a particular problem.