4 Sequential ensembles: Adaptive boosting

The ensembling strategies we’ve seen thus far have been parallel ensembles. These include homogeneous ensembles such as bagging and random forests (where the same base-learning algorithm is used to train base estimators) and heterogeneous ensemble methods such as stacking (where different base-learning algorithms are used to train base estimators).

Now, we’ll explore a new family of ensemble methods: sequential ensembles. Unlike parallel ensembles, which exploit the independence of each base estimator, sequential ensembles exploit the dependence of base estimators. More specifically, during learning, sequential ensembles train a new base estimator in such a manner that it minimizes mistakes made by the base estimator trained in the previous step.

The first sequential ensemble method we’ll investigate is boosting. Boosting aims to combine weak learners, or simple base estimators. Put another way, boosting literally aims to boost the performance of a collection of weak learners.

4.1 Sequential ensembles of weak learners

4.2 AdaBoost: Adaptive boosting

4.2.1 Intuition: Learning with weighted examples

4.2.2 Implementing AdaBoost

4.2.3 AdaBoost with scikit-learn

4.3 AdaBoost in practice

4.3.1 Learning rate

4.3.2 Early stopping and pruning

4.4 Case study: Handwritten digit classification

4.4.1 Dimensionality reduction with t-SNE

4.4.2 Boosting

4.5 LogitBoost: Boosting with the logistic loss

4.5.1 Logistic vs. exponential loss functions