A base learner, also called a base estimator, base model, or component model, is an individual predictive model whose prediction contributes to an ensemble’s final prediction. The ensemble improves on a single model by combining multiple component outputs, relying on both diversity among learners and an aggregation rule. section 1.5 section 1.6
Base learners need not all be alike. A heterogeneous ensemble can train different algorithms on the same data—for example, kernel ridge regression, support vector regression, decision-tree regression, k-nearest neighbors, Gaussian processes, and neural networks—then average their regression predictions. Diversity matters because differently behaving models can offset one another’s errors. section 1.4 section 3.0
Figure 1.7 Our first ensemble method ensembles the predictions of six different regression models by averaging them. This simple ensemble illustrates two key principles of ensembling: (1) model diversity, achieved in this case by using six different base machine-learning models; and (2) model aggregation, achieved in this case by simple averaging across predictions.
Six regression base learners make separate predictions; their mean becomes the ensemble prediction. The diagram distinguishes the individual models from the aggregation step that turns their outputs into one result. section 1.4
How base learners are used depends on the ensemble design:
- In homogeneous parallel ensembles, each learner uses the same algorithm, often a decision tree, but trains independently on a different bootstrap sample. Sampling with replacement changes the examples each learner sees and creates diversity. section 2.2.1 section 2.2.2
- In heterogeneous parallel ensembles, learners use different algorithms and train independently; their predictions can be combined by a hand-defined weighting rule or passed to a second-level meta-estimator in stacking. section 3.0 section 3.5
- In sequential ensembles such as AdaBoost, each new weak learner responds to earlier mistakes. Higher weights on misclassified training examples steer the next learner, while better-performing learners receive greater influence in the final weighted prediction. section 4.2.2 section 4.2.1
Learner strength is relative to task performance and complexity. A deep decision tree with good predictive performance is a strong learner; a shallow tree that performs only slightly better than chance is a weak learner. Parallel methods commonly use strong learners, whereas AdaBoost commonly uses weak decision stumps. section 1.5 section 4.2.2
From unstable trees to a bagged classifier
Train many depth-12 decision trees on bootstrap samples of noisy two-class moon data, then let the ensemble take the majority prediction. A single tree scored 83.84% test accuracy, while a 500-tree bagging ensemble scored 89.90%; aggregation smooths individual overfitting and sensitivity to outliers. section 2.2.2
Base learners can also be reused as existing models. This makes heterogeneous ensembles practical when independently developed predictors, such as an SVM and logistic regression model, are available. The important design question is not merely how many learners to include, but whether their errors and predictions are sufficiently diverse for aggregation to help. section 3.0 section 1.6
