Bagging

As explained in Ensemble Methods for Machine Learning

Bagging is an ensemble technique that trains multiple models independently and combines their predictions.

Bagging, short for bootstrap aggregating, turns one unstable learner into a more robust ensemble. Rather than trust a single decision tree that may overfit or react strongly to outliers, train many copies on slightly different versions of the training data and combine their answers. The models use the same learning algorithm, but randomized samples give them useful diversity. section 2.2 section 2.2.2

Figure 2.2 Bagging, illustrated. Bagging uses bootstrap sampling to generate similar but not exactly identical subsets (observe the replicates here) from a single data set. Models are trained on each of these subsets, resulting in similar but not exactly identical base estimators. Given a test example, the individual base-estimator predictions are aggregated into a final ensemble prediction. Also observe that training examples may repeat in the replicated subsets; this is a consequence of bootstrap sampling.

Use the diagram to follow the full flow: sample training examples with replacement, fit one base estimator per replicate, then aggregate their predictions for a test example. Repeated examples within a replicate are expected. section 2.2

Bagging has two independent stages:

• Create a bootstrap sample for each estimator by drawing training examples with replacement. Each replicate differs from the original and from the others, so the fitted estimators differ too. section 2.2 section 2.2.2
• Fit every base estimator independently. This makes the training work naturally parallelizable across CPU cores or other computing resources. section 2.2.2 section 2.2.4
• Aggregate predictions: use majority voting for classification and the mean for regression. Aggregation smooths individual mistakes into one ensemble prediction. section 2.2 section 7.2.1

Bagging is especially useful for complex, nonlinear, unstable estimators that overfit in different places when trained on different bootstrap samples. Aggregation reduces that variation and often produces smoother decision boundaries. It is less attractive when training many fully fitted models exceeds the available compute budget, although parallel execution can reduce elapsed training time. section 2.2.3 section 7.2.1

Bagged decision trees on noisy two-dimensional data

A simple implementation repeatedly samples rows, fits decision trees, and takes the mode of their class predictions. On a noisy two-dimensional data set, 500 depth-12 trees trained on bootstrap samples of 300 examples achieved 89.90% test accuracy, compared with 83.84% for one tree. In scikit-learn, BaggingClassifier exposes the base estimator, n_estimators, max_samples, oob_score, and n_jobs; out-of-bag accuracy estimates generalization using examples omitted from each tree’s bootstrap sample. section 2.2.2 section 2.2.3 section 2.2.4

Ensemble size and base-learner complexity are practical tuning questions. In a breast-cancer diagnosis experiment, training and test error decreased as the number of estimators grew, then test error stabilized; bagging, random forests, and Extra Trees all outperformed a single tree. Random forests extend bagging with randomized decision trees, while related variants change whether examples, features, or both are sampled. section 2.5.2 section 2.6

How Ensemble Methods for Machine Learning covers it

How deep it goes

Works through sampling, fitting, voting, averaging, from-scratch trees, scikit-learn use, OOB evaluation, and parallelization.

Examples

Uses noisy two-dimensional classification, scikit-learn OOB scoring, parallel CPU training, and Wisconsin breast-cancer diagnosis evaluation.

What it assumes

Assumes familiarity with training data, classifiers, regression, decision trees, test sets, and generalization.

What you can do after

Apply it via scikit-learn BaggingClassifier, including OOB scoring and parallel execution, or build a basic tree ensemble.

How it connects

Connects bootstrap sampling and model aggregation with decision trees, random forests, Extra Trees, pasting, random subspaces, and random patches.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningAn ensemble technique that trains successive learners on random bootstrap subsets of data and aggregates their predictions.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA collection of models trained on resampled data whose predictions are aggregated to reduce variance.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthBagging is an ensemble technique that trains models on bootstrap-resampled data and aggregates their predictions to reduce variance.

Compare how 4 books explain bagging →

Related concepts

• random forestused by · A random forest is an ensemble model that combines predictions from multiple decision trees built with randomized data samples, feature selections, or both.
• bootstrap samplinguses · Bootstrap sampling is a resampling technique that repeatedly draws observations with replacement from observed data to estimate a statistic’s sampling behavior or variability.
• modeuses · The statistical mode is the value or category occurring most frequently in a set of observations or outcomes.
• random subspacesincludes · Random subspaces is an ensemble technique that trains each model on a randomly selected subset of input features.
• data aggregationuses · Data aggregation is the combination of multiple values, records, observations, or results into a consolidated or summarized output.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.