liveBook · Manning

Out-of-bag observation

As explained in Timeless Algorithms, The Foundational Ideas

An out-of-bag observation is a training observation omitted from a particular bootstrap sample and evaluated using trees that did not include it.

An out-of-bag observation is a training case omitted from a tree’s bootstrap sample. With N draws from N observations, a particular case is omitted with probability (1 − 1/N)^N, which approaches 36.8% as N grows; about 63.2% appear at least once. section 9.2.1

For each training case, predict using only trees whose bootstrap samples omitted it. Combine those predictions by majority vote for classification or averaging for regression, then compare with the known outcome. The resulting predictions support error estimates and other performance summaries; they also provide a basis for permutation importance. Out-of-bag evaluation is internal validation, not a reason to forgo an independent test set after model selection and tuning. section 9.5 section 9.6 section 9.9

A 500-tree forest

A case is omitted from about 36.8% of 500 independently sampled trees—roughly 184, though the actual count varies. Those trees alone contribute its out-of-bag prediction. section 9.5

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Trace omitted-case predictions through trees, aggregate votes or averages, and calculate error metrics; retain an independent test set after tuning.
Examples
Illustrates the approximate 184 eligible trees for one case in a 500-tree forest.
What it assumes
Assumes bootstrap samples drawn from the training observations and teaches how their omitted cases support evaluation.
What you can do after
Evaluate forest predictions by generating out-of-bag estimates and comparing them with known outcomes.
How it connects
Connects bootstrap sampling with random forests, internal validation, classification, regression, and permutation importance.

Also explained in

Compare how 2 books explain out-of-bag observation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.