Out-of-bag observation

As explained in Ensemble Methods for Machine Learning

An out-of-bag sample is the subset of original training examples omitted from a particular bootstrap sample.

An out-of-bag (OOB) observation is a training example omitted from one bootstrap sample. Because its corresponding estimator never sees it during learning, it acts like held-out data. section 2.2.1

Bootstrap sampling draws examples with replacement, so some records repeat while others are selected zero times. Each replicate therefore splits the data into a bootstrap sample for training and an OOB sample for evaluation. section 2.2.1

For each base estimator, the procedure is:

• Draw a same-sized bootstrap sample with replacement. section 2.2.1
• Mark examples never selected as that estimator's OOB observations. section 2.2.1
• Train on the bootstrap sample and use the omitted observations to estimate performance without a separate validation set or cross-validation. section 2.2.1

A 50-record bootstrap sample

Sampling patient-record IDs 0–49 with replacement produces repeated IDs and leaves IDs such as 0, 2, 5, and 10 unselected. Those unselected records form the OOB sample for that replicate. section 2.2.1

Figure 2.3 Bootstrap sampling illustrated on a data set of six examples. By sampling with replacement, we can get a bootstrap sample size of six, containing only four unique objects but with repeats. Performing bootstrap sampling several times produces several replicates of the original data set—all of them with repeats.

Visualize how a six-example dataset produces repeated bootstrap entries and a complementary set of omitted observations. section 2.2.1

With scikit-learn, setting oob_score=True on BaggingClassifier enables OOB evaluation. The resulting OOB score estimates ensemble generalization and can be compared with performance on a separately held-out test set. section 2.2.3

How Ensemble Methods for Machine Learning covers it

How deep it goes

Explains bootstrap partitioning, gives NumPy construction steps, and connects OOB evaluation to BaggingClassifier usage.

Examples

Demonstrates omitted patient-record IDs in NumPy and compares a scikit-learn OOB score with held-out test accuracy.

What it assumes

Assumes familiarity with training examples and model evaluation, while defining sampling with replacement and bootstrap samples operationally.

What you can do after

Apply it via NumPy and scikit-learn to generate OOB sets and estimate bagging performance.

How it connects

Connects bootstrap sampling to bagging, ensemble generalization, validation sets, cross-validation, and test-set evaluation.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Related concepts

• bootstrap samplingproduced by · Bootstrap sampling is a resampling technique that repeatedly draws observations with replacement from observed data to estimate a statistic’s sampling behavior or variability.
• evaluation datasetequivalent to · An evaluation dataset is a collection of examples or test cases, often with expected or judged outcomes, designated to assess a model or system against defined criteria.
• random forestused by · A random forest is an ensemble model that combines predictions from multiple decision trees built with randomized data samples, feature selections, or both.
• out-of-bag errorused by · Out-of-bag error is an estimate of an ensemble’s prediction error computed for each observation using only ensemble members whose bootstrap samples omitted it.
• accuracymeasures · Accuracy is the degree to which an output, prediction, or measurement agrees with a correct or reference outcome under a specified criterion.
• importance scoreused by · An importance score is a numerical value indicating an item's relative significance, relevance, or contribution within a specified context.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.