Extrapolation estimates target values beyond the range observed during training. It matters when future cases, especially time-series observations, may differ substantially from the past. Gradient boosting can extrapolate because it adds the outputs of a sequence of models, unlike random forests, whose regression predictions remain based on means of training subsamples. section 5.2.2
Choose an approach according to how far the prediction must extend:
- Use gradient-boosted decision trees when moderate extrapolation is useful, including forecasting future values beyond the training targets. section 5.2.2
- Prefer a linear model when inputs are far outside the training range or extreme outliers require stronger extrapolative behavior. section 5.2.2
- Do not expect decision-tree regression or random-forest ensembles to predict target values above or below those seen during training. section 5.2.2
Gradient boosting regression
A regression variant removes classification logit and sigmoid transformations, initializes predictions with the mean target, then repeatedly fits trees to gradients and adds their weighted predictions. The resulting predictions can exceed the 0–1 range, demonstrating extrapolation beyond learned target limits. section 5.2.2
Listing 5.8 Plotting gradient boosting predicted probabilities
import matplotlib.pyplot as plt
proba = cls.predict_proba(Xt) #1
plt.figure(figsize=(8, 6))
plt.hist(proba,
bins=30,
density=True,
color='blue',
alpha=0.7) #2
plt.xlabel('Predicted Probabilities')
plt.ylabel('Density')
plt.title('Histogram of Predicted Probabilities')
plt.grid(True)
plt.show()
Plot predicted probabilities with a histogram to inspect their distribution before comparing regression behavior. section 5.2.2
