A meta-learner is the decision-maker at the top of an ensemble. Rather than relying on one model, it learns how to turn several base-estimator outputs into a final prediction. The benefit comes from diversity: models that make different errors can preserve shared signal when their outputs are combined. section 1.1 section 1.6
In stacking, base estimators form level 1. Their predictions or prediction probabilities become meta-features—one or more new features for each example. A level-2 estimator, the meta-learner, trains on those meta-features and target labels to learn the combining function. This differs from weighting, where a predetermined function combines model outputs directly. section 3.0 section 3.3.1
Figure 3.11 The probability of prediction of each training example according to 3nn and gnb are used as meta-features for a new classifier. Data points in darker regions indicate high-confidence predictions. Each training example now has two meta-features, one each from 3nn and gnb.
Two base classifiers each contribute a prediction probability, creating two meta-features for the second-level classifier. section 3.3.1
A practical stacked meta-learner follows this flow:
- Train diverse level-1 base estimators, such as nearest neighbors and Gaussian naive Bayes, or a broader mix of linear and nonlinear models. section 3.3.1
- Run each base estimator on examples and assemble its predictions or probabilities into a meta-feature matrix. section 3.3.1
- Fit a level-2 estimator on that matrix and the original labels; logistic regression provides a fast linear starting point, while SVMs, neural networks, and other nonlinear models can learn more complex combinations. section 3.3.1
- For new inputs, regenerate the same meta-features with level-1 models and pass them to the trained meta-learner. section 3.3.1
Training a meta-learner on base-model outputs from the same examples used to fit those base models can overfit, especially with noisy data or a flexible second-level model. Cross-validated stacking generates meta-features from held-out folds: train each base estimator on a fold’s training portion, predict its validation portion, fit the meta-learner on the collected out-of-sample predictions, then retrain base estimators on all data. StratifiedKFold preserves class ratios across classification folds. Blending is a faster alternative that reserves one validation split for meta-feature generation, at the cost of using less data for each stage. section 3.3.2 section 3.4.3
Regression stack
For fuel-efficiency prediction, four heterogeneous regressors—kernel ridge, support vector regression, k-nearest neighbors, and Extra Trees—supply predictions to an MLPRegressor meta-learner through scikit-learn’s StackingRegressor. This uses a nonlinear second-level model to learn interactions among the base predictions; tuning the base regressors can further improve the ensemble. section 7.2.3 section 7.2
Choose the level-2 model to match the tradeoff. Linear stacking is efficient and useful for exploration, whereas nonlinear meta-learners have greater representational power but raise overfitting risk. Prediction probabilities often provide a smoother, more nuanced input than hard predictions. Evaluate the complete pipeline on held-out data, not only its training performance. section 3.5 section 3.4.3
