This chapter covers
- Understanding ensemble methods
- Using bagging, boosting, and stacking
- Using the random forest and XGBoost algorithms
- Benchmarking multiple algorithms against the same task
In the last chapter, I showed you how we can use the recursive partitioning algorithm to train decision trees that are very interpretable. We finished by highlighting an important limitation of decision trees: they have a tendency to overfit the training set. This results in models that generalize poorly to new data. As a result, individual decision trees are rarely used, but they can become extremely powerful predictors when many trees are combined together.
By the end of this chapter, you’ll understand the difference between ordinary decision trees and ensemble methods, such as random forest and gradient boosting, which combine multiple trees to make predictions. Finally, as this is the last chapter in the classification part of the book, you’ll learn what benchmarking is and how to use it to find the best-performing algorithm for a particular problem. Benchmarking is the process of letting a bunch of different learning algorithms fight it out to select the one that performs best for a particular problem.