liveBook · Manning

Out-of-bag evaluation

As explained in Timeless Algorithms, The Foundational Ideas

Out-of-bag evaluation estimates a model’s predictive performance using observations omitted from the samples used to train the models making those predictions.

A random forest can estimate how well it predicts without first setting aside a validation sample. For each training observation, it combines predictions from trees whose bootstrap samples omitted that observation. Comparing these predictions with known outcomes gives an internal estimate of performance on new data, while the full data set still contributes to forest training. section 9.5

The same out-of-bag predictions support different performance summaries:

  • For classification, calculate the error rate; also derive a confusion matrix, class-specific error rates, and out-of-bag margins. section 9.5
  • For regression, compare predicted and observed outcomes using mean squared error, mean absolute error, or related measures. section 9.5

Unlike k-fold cross-validation, out-of-bag evaluation does not divide observations into fixed groups or retrain the forest for each fold. Bootstrap sampling supplies overlapping validation sets as a byproduct of training: each observation is evaluated by the trees that did not train on it. section 9.5

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines per-observation predictions from trees that omitted it, gives classification and regression metrics, and contrasts the mechanics with k-fold cross-validation.
Examples
Uses classification error, confusion matrices, class-specific errors, margins, mean squared error, and mean absolute error.
What you can do after
Evaluate predictive performance using omitted-observation predictions and classification or regression measures.
How it connects
Connects bootstrap sampling and random forests with k-fold cross-validation, confusion matrices, margins, and regression error measures.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.