A random forest can estimate how well it predicts without first setting aside a validation sample. For each training observation, it combines predictions from trees whose bootstrap samples omitted that observation. Comparing these predictions with known outcomes gives an internal estimate of performance on new data, while the full data set still contributes to forest training. section 9.5
The same out-of-bag predictions support different performance summaries:
- For classification, calculate the error rate; also derive a confusion matrix, class-specific error rates, and out-of-bag margins. section 9.5
- For regression, compare predicted and observed outcomes using mean squared error, mean absolute error, or related measures. section 9.5
Unlike k-fold cross-validation, out-of-bag evaluation does not divide observations into fixed groups or retrain the forest for each fold. Bootstrap sampling supplies overlapping validation sets as a byproduct of training: each observation is evaluated by the trees that did not train on it. section 9.5
