A residual error measures the gap between a target and a model’s current prediction. Its magnitude shows how badly that example is fit; in regression, its sign also shows the direction of the mistake: positive means the prediction is too low, while negative means it is too high. Residuals turn a model’s remaining mistakes into a training signal for the next stage. section 5.2.1 section 7.3
With squared loss, the negative loss gradient is the residual itself. Gradient boosting therefore fits a weak regressor to residuals rather than directly changing example weights. Large residuals receive more attention because they represent larger gaps between predictions and labels. section 5.2.1 section 5.0
A residual-driven boosting iteration proceeds as follows:
- Start with a current ensemble prediction, often a constant such as zero. section 5.2.2
- Compute a residual for every training example from the current prediction and loss. section 5.2.2
- Fit a weak regression tree to predict those residuals from the features. section 5.2.2
- Choose a step length, then add the weighted tree to the ensemble prediction. section 5.2.2
Residuals connect gradient boosting to gradient descent: gradient descent adds a negative-gradient vector directly, while gradient boosting approximates that direction with a learned weak regressor. They also distinguish gradient boosting from AdaBoost, which emphasizes difficult examples through higher weights instead of fitting their residual values. section 5.2.1 section 5.2.2
Two-moons classification
For labels converted from 0/1 to −1/1 and initial predictions of zero, Class 1 examples begin with residual +1 and Class 0 examples with residual −1. A shallow regression tree learns this first residual pattern; later trees target what the accumulated ensemble still gets wrong. A 10-estimator implementation reports 6% test error on a noisy two-moons split. section 5.2.2
Figure 5.8 Comparing AdaBoost (left) to gradient boosting (right). Both approaches train weak estimators that improve classification performance on misclassified examples. AdaBoost uses weights, with misclassified examples being assigned higher weights. Gradient boosting uses residuals, with misclassified examples having higher residuals. The residuals are nothing but negative loss gradients.
Compare two ways to focus later learners: AdaBoost raises weights on misclassified examples, whereas gradient boosting supplies residuals, the negative loss gradients. section 5.2.1
Because residuals derive from the selected loss, changing the loss changes which errors receive priority. This lets gradient boosting address classification, regression, and ranking problems. Learning rate and early stopping can limit overfitting; histogram-based tree learning, sampling, and feature bundling can improve large-scale training efficiency. section 5.0 section 5.6
