Linear blending

As explained in Ensemble Methods for Machine Learning

Linear blending is an ensemble technique that combines model outputs using a weighted linear combination.

Linear blending combines predictions from different models into one ensemble output. Its practical appeal became clear during the Netflix Prize, where teams combined many diverse models to improve accuracy. section 3.0 section 1.0

The intuition is straightforward: let several independently trained models contribute according to performance-based weights, then add their outputs as a weighted linear combination. Unlike bagging and random forests, linear blending gains diversity from different learning algorithms rather than only from resampling. section 3.0

A practical blending workflow is:

• Select diverse base learners, such as a decision tree, support vector machine, or artificial neural network. section 3.0
• Train the base estimators independently, including models that may already have been trained elsewhere. section 3.0
• Assign weights based on model performance and combine the resulting predictions linearly. section 3.0

Netflix Prize

Early Netflix Prize systems used models with different parameterizations and a linear blend. As the competition evolved, teams replaced the linear blend with a nonlinear one, illustrating both the usefulness and the limits of linear combination. section 1.0

Figure 3.1 Homogeneous ensembles (chapter 2), such as bagging and random forests, use the same learning algorithm to train base estimators, and they achieve ensemble diversity through random sampling. Heterogeneous ensembles (this chapter) use different learning algorithms to achieve ensemble diversity.

Compare linear blending’s heterogeneous model diversity with homogeneous ensembles such as bagging and random forests. section 3.0

How Ensemble Methods for Machine Learning covers it

How deep it goes

Explains performance-based weighting and heterogeneous diversity, without detailing an implementation or weight-fitting procedure.

Examples

Uses the Netflix Prize and heterogeneous combinations of decision trees, support vector machines, and artificial neural networks.

What it assumes

Assumes familiarity with base estimators and ensemble methods while distinguishing heterogeneous models from homogeneous ensembles.

What you can do after

Apply it by combining predictions from diverse existing or independently trained models with performance-based weights.

How it connects

Connects heterogeneous ensembles with stacking, bagging, random forests, cross-validation, and nonlinear blending.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Related concepts

• ensembleimplements · An ensemble is a technique that combines or considers outputs from multiple models or component methods collectively to produce or assess a joint result.
• meta-learningkind of · Meta-learning is a machine-learning approach that uses experience across tasks to improve how systems learn or adapt to new tasks.
• blendingalternative to · Blending is an ensemble technique that trains a higher-level learner using predictions generated on a single held-out validation set rather than multiple cross-validation folds.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.