Model aggregation

As explained in Ensemble Methods for Machine Learning

Model averaging is an aggregation method that combines predictions by averaging outputs from multiple models.

Model aggregation turns several model predictions into one final prediction. Its value comes from combining models that make different mistakes: the collective result can smooth individual errors and generalize better than a single model. Diversity supplies different opinions; aggregation supplies the decision rule that joins them. section 1.1 section 1.6

For regression, model averaging usually means taking the arithmetic mean of continuous predictions. Train several base estimators, ask each to predict new data, collect those outputs, and average across models for each case. In a six-model experiment, larger ensembles showed improving mean performance and decreasing variation across ensemble combinations. section 1.4

Choose the aggregation rule to match the prediction task:

• Regression: average continuous predictions, optionally with a weighted mean. section 1.4 section 2.2.1
• Classification: select the majority vote, also called the statistical mode, from base-learner predictions. section 2.2.1
• Other ensembles: use combination functions or another machine-learning model as the aggregator when appropriate. section 2.2.1

Averaging six regressors

Build a heterogeneous ensemble by training kernel ridge regression, support vector regression, a decision tree, k-nearest neighbors, a Gaussian process, and a multilayer perceptron on the same data. On test data, place each model’s prediction in one column and compute the row-wise mean. The result is one ensemble prediction per test example. section 1.4

Figure 1.7 Our first ensemble method ensembles the predictions of six different regression models by averaging them. This simple ensemble illustrates two key principles of ensembling: (1) model diversity, achieved in this case by using six different base machine-learning models; and (2) model aggregation, achieved in this case by simple averaging across predictions.

The diagram separates the two ingredients of a simple ensemble: diverse regression models create individual predictions, and averaging turns them into one output. section 1.4

Aggregation cannot guarantee a correct answer when component models share the same mistake. It is most useful alongside diversity, whether models differ by learning algorithm or, in parallel ensembles, by how they are trained from the data. For bagging-style classification, independently trained base learners commonly feed a majority-vote aggregator. section 1.1 section 2.1 section 2.2.1

How Ensemble Methods for Machine Learning covers it

How deep it goes

Explains simple regression averaging and classification majority voting, while noting weighted means, combination functions, and learned aggregators.

Examples

Demonstrates six regression algorithms whose test predictions are averaged, alongside a medical-diagnosis majority-vote analogy.

What it assumes

Assumes familiarity with regression, classification, model predictions, and training base estimators.

What you can do after

Build it from scratch by collecting base-model predictions and averaging them for regression or voting for classification.

How it connects

Connects ensemble diversity, base estimators, majority voting, bootstrap sampling, bagging, and the bias-variance tradeoff.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Related concepts

• baggingused by · Bagging, or bootstrap aggregation, is an ensemble technique that trains multiple models on bootstrap samples of data and combines their predictions.
• ensembleimplemented by · An ensemble is a technique that combines or considers outputs from multiple models or component methods collectively to produce or assess a joint result.
• modeuses · The statistical mode is the value or category occurring most frequently in a set of observations or outcomes.
• prediction averagingimplemented by · Prediction averaging is an aggregation technique that combines multiple numerical values or predictions by computing their arithmetic mean.
• forecastingproduces · Forecasting is the process of predicting future values or outcomes from information currently available, often including historical observations.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.