Heterogeneous ensemble

As explained in Ensemble Methods for Machine Learning

A heterogeneous parallel ensemble is an ensemble whose independently trained component models use different model types or learning algorithms.

A heterogeneous ensemble combines independently trained models that use different learning algorithms. Think of several specialists reaching a diagnosis: a decision tree, SVM, and neural network can disagree in useful ways, then contribute to one final prediction. Unlike homogeneous ensembles, which create diversity through random sampling with one algorithm, heterogeneous ensembles create it by varying the algorithms themselves. section 2.1 section 3.0

Figure 3.1 Homogeneous ensembles (chapter 2), such as bagging and random forests, use the same learning algorithm to train base estimators, and they achieve ensemble diversity through random sampling. Heterogeneous ensembles (this chapter) use different learning algorithms to achieve ensemble diversity.

Compare the two routes to diversity: resampling for homogeneous ensembles versus different learning algorithms for heterogeneous ensembles. section 3.0

The workflow has two parts: fit each base estimator separately, then aggregate its predictions. For classification, aggregation can use hard class labels or soft prediction probabilities; probabilities typically give smoother output. Better models may receive larger performance-based weights, or a learned meta-estimator can determine how to combine model outputs. section 3.1.2 section 3.5

Common ways to combine diverse models include:

• Weighted combination—assign each base estimator a weight based on performance, then use a predefined function such as a weighted sum. section 3.5
• Stacking—train first-level estimators, then train a second-level meta-estimator on their outputs. section 3.5
• Blending—use one validation split to generate meta-features for the second-level estimator instead of cross-validation. section 3.4.3
• Regression aggregation—combine continuous predictions with mean, median, minimum, or maximum; median can reduce an outlier prediction's influence. section 7.2.2

Blending text classifiers

Split training data into training and validation portions. Fit a random forest, Extra Trees, logistic regression, Bernoulli naïve Bayes, and linear SVM on the training portion. Their validation predictions become five meta-features, appended to the original features, for a neural-network meta-estimator. Mixing linear and nonlinear classifiers supplies the intended diversity. section 3.4.3

Diversity alone does not guarantee improvement. Complex nonlinear combination functions and stacking can overfit, especially with noisy data. Cross-validation reduces this risk by separating the data used for base-model and meta-model predictions, but increases training time; blending is a faster validation-set alternative. section 3.5

How Ensemble Methods for Machine Learning covers it

How deep it goes

Develops training, hard versus soft predictions, weighting, stacking, blending, regression aggregators, and overfitting tradeoffs.

Examples

Uses a 2D two-class data set, text classifiers, and AutoMPG fuel-efficiency regression predictions.

What it assumes

Assumes familiarity with supervised classifiers or regressors, train-validation-test splits, and prediction probabilities.

What you can do after

Apply it via scikit-learn by fitting diverse estimators and combining labels or probabilities with aggregators or a meta-estimator.

How it connects

Contrasts homogeneous ensembles, bagging, random forests, stacking, blending, cross-validation, and meta-learning.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Related concepts

• base-learning algorithmuses · A base-learning algorithm is an algorithm used to train an individual base estimator within an ensemble.
• diversityproduces · Diversity is the degree of variation among members of a set or population in relevant characteristics, perspectives, behaviors, or outcomes.
• meta-learninguses · Meta-learning is a machine-learning approach that uses experience across tasks to improve how systems learn or adapt to new tasks.
• parallel ensemble methodkind of · A parallel ensemble method combines outputs from multiple independently trained base learners that perform the same task.
• base learnerhas part · A base learner is an individual predictive model whose output contributes to an ensemble or higher-level estimator.
• meta-learneruses · A meta-learner is a second-level learning model that combines predictions produced by base estimators to make final predictions.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.