Ensemble diversity is the useful difference among an ensemble’s component models: they should bring different opinions to a prediction rather than repeat the same mistake. Aggregating varied predictions can cancel some individual errors, making the combined model more robust and better able to generalize than a single estimator. section 1.1 section 1.4
Figure 1.7 Our first ensemble method ensembles the predictions of six different regression models by averaging them. This simple ensemble illustrates two key principles of ensembling: (1) model diversity, achieved in this case by using six different base machine-learning models; and (2) model aggregation, achieved in this case by simple averaging across predictions.
Six regression models make separate predictions, then their predictions are averaged. The example separates diversity in the base models from aggregation in the final prediction. section 1.4
Diversity and model aggregation are complementary ingredients. Diversity supplies distinct errors and perspectives; aggregation turns those separate outputs into one prediction, such as a majority decision for classification or a mean for regression. Agreement is common on easy cases, while difficult cases expose more variation among individual models. section 1.1 section 1.4
Parallel ensembles create diversity in two main ways:
- Homogeneous ensembles keep one learning algorithm but vary its training data. Bootstrap sampling draws same-size samples with replacement, so each base estimator sees a different pattern of repeated and omitted examples. section 2.2.1 section 7.2
- Heterogeneous ensembles keep the data fixed but use different base-learning algorithms, such as a decision tree, support vector machine, and neural network. section 3.0 section 7.2
- Variants including pasting, random subspaces, random patches, and Extra Trees introduce randomization in different ways to ensure diversity. section 2.4
A simple diverse regression ensemble
Train Kernel Ridge, support vector, decision-tree, nearest-neighbor, Gaussian-process, and neural-network regressors on a generated regression data set. Collect each test prediction in a matrix and average across models. As combinations grow from one model to all six, mean performance increases and variation among ensemble combinations decreases. section 1.4 section 1.4
Diversity is not guaranteed merely by having many models. An effective ensemble needs estimators that are genuinely different enough not to fail together. In bagging-style methods, resampling creates those differences; in heterogeneous methods, algorithm choice does. Stacking can further use cross-validation so base estimators see different data subsets, though it can increase training time and may overfit on noisy data. section 2.1 section 3.5
From competition blends to regression
Large recommendation-system ensembles combined many diverse models and features to improve predictions beyond individual approaches. The same principle applies to continuous labels: independently trained regression estimators can be fused into a joint prediction, such as predicting vehicle fuel efficiency from engine attributes. section 1.0 section 7.2
