A weak regressor is a regression model with limited predictive strength that learns from the current residuals and contributes one additive update to a boosting ensemble. Decision-tree regression commonly supplies the base learner. section 5.2.1
The intuition is corrective: after the ensemble makes predictions, residuals identify what remains poorly explained. The next regressor fits those real-valued residuals, approximating the negative loss gradient. Its weighted prediction is then added to the current model. section 5.2.1
A boosting iteration proceeds as follows:
- Compute each example’s negative residual from the current model. section 5.2.2
- Fit a weak decision-tree regressor to the examples and their residuals. section 5.2.1 section 5.2.2
- Find a step length and update the ensemble with the regressor’s scaled predictions. section 5.2.1 section 5.2.2
Two-dimensional classification
On a noisy two-moons dataset, shallow regression trees are trained sequentially even though the final task is classification. The first tree reduces the error to 16%; after two trees, ensemble accuracy reaches 91%; after ten, training accuracy reaches 97.5%. A from-scratch implementation reports a 6% test error. section 5.2.2
initialize: F = f0, some constant value for t = 1 to T: 1. compute the negative residuals for each example, rti = (∂L/∂F)(xi) 2. fit a weak decision tree regressor ht(x) using the training set (xi, ri)ni=1 3. compute the step length (αt) using line search 4. update the model: Ft = F + αt ⋅ ht(x)
Use the procedural listing to follow residual computation, regressor fitting, line search, and additive model updates in order. section 5.2.2
