Ensemble

As explained in Ensemble Methods for Machine Learning

An ensemble is a predictive model formed by combining the outputs of multiple component models.

An ensemble is a predictive meta-estimator that combines the outputs of multiple base models, also called base learners or base estimators. The intuition is the wisdom of crowds: several capable predictors are less likely to make exactly the same mistake, especially when their strengths differ. Like a diagnostic team that votes on a case, the ensemble turns many opinions into one final prediction. section 1.1 section 1.5 section 1.6

Ensembling is useful when a single model faces the fit-versus-complexity, or bias-variance, tradeoff. Simple models may underfit; highly complex models may fit training data very well yet overfit and generalize poorly. Combining several low-bias models can reduce variance, while combining several low-variance models can reduce bias. The aim is better performance on future data rather than merely a better training fit. section 1.3 section 1.3.2 section 1.6

An effective ensemble needs two ingredients:

• Diversity: base models should make different errors. Diversity can come from using different learning algorithms, different training samples, or dependent training procedures. section 1.4 section 1.5 section 2.2.1
• Aggregation: combine the base predictions into one output. Classification can use the most common prediction; regression can use the mean of continuous predictions. section 1.1 section 1.4

A six-model regression ensemble

Train kernel ridge regression, support vector regression, a decision tree, k-nearest neighbors, a Gaussian process, and a multilayer perceptron on the same Friedman-1 data. For each new example, collect the six predictions and take their mean. Comparing every combination from one through six models shows a practical pattern: larger ensembles have higher mean performance and lower variation across combinations. Diversity matters more than simply adding copies of the same predictor. section 1.4 section 1.4

Ensembles are commonly organized by how their components are trained. Parallel ensembles train base models independently and can run concurrently. They may be homogeneous, using one model family such as decision trees, or heterogeneous, using different algorithms. Bagging, random forests, and Extra Trees are homogeneous parallel methods; stacking is a heterogeneous parallel method. Sequential ensembles instead make later learners depend on earlier ones. section 1.5

Figure 1.7 Our first ensemble method ensembles the predictions of six different regression models by averaging them. This simple ensemble illustrates two key principles of ensembling: (1) model diversity, achieved in this case by using six different base machine-learning models; and (2) model aggregation, achieved in this case by simple averaging across predictions.

This diagram makes the two core mechanics visible: six different regressors supply predictions, and a simple average produces the ensemble prediction. section 1.4

For homogeneous ensembles, bootstrap sampling provides a direct route to diversity. Sample the training data with replacement to make multiple same-size replicas. Train one base estimator per replica, then aggregate their predictions. Examples absent from a particular bootstrap sample form its out-of-bag sample, which can act as held-out data for evaluating the ensemble without a separate validation set or cross-validation procedure. section 2.2.1 section 2.2.1

Ensembles are especially prominent for structured data, where feature engineering combined with methods such as XGBoost, random forests, and gradient boosting has performed strongly in competitions. They also support practical work in finance, health care, cybersecurity, education, manufacturing, recommendation systems, and entertainment. For categorical demographic data, preprocessing can feed scikit-learn random forests, LightGBM, and XGBoost, while CatBoost can handle categorical features during training. section 1.2 section 8.3

More models are not an automatic guarantee of success: the benefit rests on useful diversity and sound evaluation. Tune ensemble size and base-estimator complexity, compare against single-model baselines, and evaluate on held-out data or with cross-validation. In bike-rental regression, grid search with cross-validation tunes random forest and Extra Trees over `n_estimators` and `max_depth`, then compares loss functions and test metrics with stacking and individual generalized linear models. section 7.4.3 section 1.4

How Ensemble Methods for Machine Learning covers it

How deep it goes

Defines diversity and aggregation, implements averaged regression predictions, and classifies parallel versus sequential ensembles without detailing every method.

Examples

Uses a diagnostic voting team, Netflix Prize blending, six-regressor Friedman-1 averaging, bike rentals, and demographic income prediction.

What it assumes

Assumes supervised learning, regression, classification, training-test evaluation, and model complexity while reinforcing underfitting, overfitting, and bias-variance.

What you can do after

Apply it via scikit-learn by training diverse regressors, averaging predictions, tuning parallel ensembles, and comparing held-out performance.

How it connects

Connects ensembles to bias-variance tradeoff, bagging, random forests, Extra Trees, stacking, boosting, bootstrap sampling, and out-of-bag evaluation.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Outlier Detection in PythonAn ensemble is a combination of multiple models or detectors whose outputs are combined to obtain more robust results than an individual model or detector.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA technique that combines multiple models or their predictions to improve predictive results.
• 

  liveBookmanning.com
  Statistics Every Programmer NeedsAn ensemble model combines predictions from multiple constituent models to produce a collective prediction.
• 

  liveBookmanning.com
  Experimentation for EngineersA model ensemble is a collection of predictive models whose predictions or decisions are combined, compared, or otherwise used collectively.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA technique that combines multiple models or views to improve performance by exploiting complementary errors or representations.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthAn ensemble method combines predictions or decisions from multiple models to obtain a joint predictive result.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA model ensemble is a collection of predictive models whose outputs are combined to produce a joint prediction.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA collection of multiple models whose predictions are combined, typically to improve accuracy or robustness.
• 

  liveBookmanning.com
  Generative AI in ActionA model ensemble is an architecture that combines multiple machine-learning models to improve overall capability, performance, or reliability.
• 

  liveBookmanning.com
  Building LLM Applications with DSPyA modeling technique that combines outputs from multiple models or programs.

Compare how 11 books explain ensemble →

Related concepts

• random forestincludes · A random forest is an ensemble model that combines predictions from multiple decision trees built with randomized data samples, feature selections, or both.
• gradient boostingincludes · Gradient boosting is an ensemble-learning technique that sequentially adds predictive models, typically weak learners, to reduce a loss by fitting its gradient.
• baggingincludes · Bagging, or bootstrap aggregation, is an ensemble technique that trains multiple models on bootstrap samples of data and combines their predictions.
• language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• decision treeuses · A decision tree is a branching structure that represents sequential tests or choices, with paths leading to outcomes, predictions, or actions.
• diversityuses · Diversity is the degree of variation among members of a set or population in relevant characteristics, perspectives, behaviors, or outcomes.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.