An ensemble combines multiple machine-learning models into one prediction. For tabular data, tree-based ensembles are especially common because they perform well with relatively little preprocessing. The practical goal is to improve on a single model by pooling several predictions or by building models that learn in sequence from earlier results and errors. section 5.1 section 5.1.1
Two core strategies organize most ensemble approaches:
- Averaging combines predictions from multiple models. Regression predictions are averaged; classification predictions can use majority voting. Pasting, bagging, random subspaces, and random patches create model diversity through different samples of rows, features, or both. section 5.1.1
- Boosting builds chained models and forms a weighted prediction from them. Each model is built using the results of previous models; gradient boosting, including XGBoost and LightGBM, is a prominent example. section 5.1.1 section 5.0
Averaging can make predictions more reliable by reducing variance, though sampling may slightly increase bias and can be computationally intensive. Random forests combine decision trees using bootstrapped training samples and subsampled features, producing trees that differ from one another. Their key tuning choices include the number of trees, features considered for splits, tree depth, and minimum terminal-branch size. section 5.6 section 5.1.1
Table 5.1 Bagging and sampling strategies (view table figure)
| Averaging strategy |
What happens to data |
Parameters for BaggingClassifier/BaggingRegressor |
|---|---|---|
| Pasting |
Training examples are sampled without replacement |
bootstrap = False max_samples < 1.0 max_features = 1.0 |
| Bagging |
Training examples are sampled with replacement (bootstrapping) |
bootstrap = True max_samples = 1.0 max_features = 1.0 |
| Random subspaces |
Features are sampled (without replacement) |
bootstrap = False max_samples = 1.0 max_features < 1.0 |
| Random patches |
Training examples and features are sampled without replacement |
bootstrap = False max_samples < 1.0 max_features < 1.0 |
Compare how pasting, bagging, random subspaces, and random patches vary row and feature sampling, along with their BaggingClassifier and BaggingRegressor settings. section 5.1.1
Blending XGBoost and deep learning for Airbnb prices
For Tokyo Airbnb price prediction, XGBoost outperformed fastai when evaluated alone, but their predictions showed different error patterns. Blending out-of-fold predictions at several fixed weights improved every reported metric over XGBoost alone: a 75/25 XGBoost/fastai blend reduced RMSE and MAE, while a 50/50 blend gave the best R-squared result. This illustrates why models that capture different patterns can be stronger together than separately. section 12.4 section 12.5
Choose an ensemble when a single model leaves useful predictive signal on the table and candidate models make meaningfully different errors. Test blends against standalone baselines using the metrics that matter for the task; a blend is not automatically best, and the preferred weighting can differ between R-squared and error measures such as RMSE or MAE. section 12.5 section 12.6
