Out-of-bag error

As explained in Ensemble Methods for Machine Learning

An out-of-bag score is an ensemble performance estimate computed from predictions on examples omitted from the corresponding component training samples.

When an ensemble trains each component on a bootstrap sample, some training examples are omitted from that component. Those omitted examples form its out-of-bag (OOB) sample: an effectively held-out test set for evaluating the component and, collectively, the ensemble. section 2.2.1

Compute an OOB estimate through repeated bootstrap training:

• Draw a bootstrap sample with replacement for each base estimator; repeated examples remain in training, while omitted examples become that estimator’s OOB sample. section 2.2.1
• Predict each example using only ensemble components for which that example was out of bag. section 2.2.1
• Aggregate those predictions and calculate the resulting error or score. This uses the training data efficiently without requiring a separate validation set or cross-validation procedure. section 2.2.1

A small bootstrap sample

For 50 training examples, sample 50 indices with replacement. The resulting bootstrap sample may repeat indices such as 12, 14, and 21; the remaining indices, such as 0, 2, and 5, form the corresponding OOB sample. With scikit-learn, set oob_score=True on BaggingClassifier, fit the ensemble, and read bag_ens.oob_score_. One demonstrated OOB accuracy is 96.6%, compared with 95.2% on a held-out test set. section 2.2.1 section 2.2.3

Figure 2.3 Bootstrap sampling illustrated on a data set of six examples. By sampling with replacement, we can get a bootstrap sample size of six, containing only four unique objects but with repeats. Performing bootstrap sampling several times produces several replicates of the original data set—all of them with repeats.

Visualize how sampling with replacement creates repeated training examples and a complementary OOB subset; repeating the process produces varied ensemble training sets. section 2.2.1

How Ensemble Methods for Machine Learning covers it

How deep it goes

Explains bootstrap/OOB set construction, prediction aggregation, and library retrieval, but stops short of deriving the estimator statistically.

Examples

Uses 50-index bootstrap samples, explicit OOB arrays, BaggingClassifier accuracy, and a random-forest configuration.

What it assumes

Assumes familiarity with bootstrap sampling, base estimators, and ensemble aggregation while reinforcing how held-out subsets arise.

What you can do after

Operate/configure scikit-learn ensembles with oob_score=True and interpret their OOB performance estimates.

How it connects

Connects bootstrap sampling, bagging, ensemble aggregation, validation sets, cross-validation, test sets, and random forests.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Related concepts

• BaggingClassifierused by · BaggingClassifier is a scikit-learn ensemble estimator that trains base estimators on bootstrap samples and aggregates their predictions.
• permutation feature importanceused by · Permutation feature importance measures a feature’s importance to a model by randomly permuting its values and assessing the resulting change in predictive performance.
• generalizationmeasures · Generalization is the extension of a pattern, model, solution, or conclusion beyond the specific examples or cases from which it was derived.
• bootstrap samplinguses · Bootstrap sampling is a resampling technique that repeatedly draws observations with replacement from observed data to estimate a statistic’s sampling behavior or variability.
• out-of-bag observationuses · An out-of-bag observation is a training example omitted from the bootstrap sample used to train a particular ensemble member.
• errorkind of · Prediction error is the discrepancy between a model’s predicted outcome and the corresponding actual or observed outcome.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.