A decision stump makes one decision and returns one of two leaf predictions. It is a depth-one decision tree: a single feature condition routes each example left or right, where classification returns a class label and regression returns a value. This minimal structure is easy to interpret but represents only simple functions. section 1.3.1
As a weak learner in boosting, a stump is trained repeatedly to focus on changing priorities:
- Start by giving every training example the same weight. section 4.2.1
- Train a stump that prioritizes examples according to their weights. section 4.2.1
- Increase the weights of misclassified examples so the next stump concentrates on previous mistakes. section 4.2.1
- Combine several different stumps into a stronger ensemble. section 4.1 section 4.2.1
Three rounds of boosting
Suppose the first stump misclassifies 15% of the training examples. Those mistakes receive larger weights, and a second stump is trained to prioritize them. After the second stump makes different mistakes, a third stump focuses on those examples. The three stumps can then be combined into a stronger learner, even though each individual stump is only slightly better than random guessing. section 4.2.1 section 4.1
