How much does a trained forest rely on a feature? Compare its baseline out-of-bag error with the error after shuffling that feature’s values. For feature j, define importance as ΔIⱼ = Eperm,j − Ebase. A positive score means error increased when the feature’s information was disrupted. section 9.6.1
The procedure keeps the forest fixed while disrupting one feature at a time:
- Measure baseline out-of-bag error, using classification error, mean squared error, or another suitable measure. section 9.6.1
- Shuffle one feature’s values among out-of-bag observations, leaving other features unchanged; recalculate predictions with the same trained trees. section 9.6.1
- Subtract baseline error from permuted error. In Breiman’s procedure, average the increase across each tree’s out-of-bag observations and then across the forest. section 9.6.1
Redundant temperature measures
If a dataset contains temperature in both Celsius and Fahrenheit, either feature can substitute for the other. Shuffling one may barely change error, so its score can be low even though temperature is predictive. Interpret importance across related features when information is redundant. section 9.6.2
A low score can reflect redundancy rather than irrelevance. Shuffling correlated features may also create predictor combinations rarely seen in real data. Importance measures predictive reliance given the other features—not causality; a predictive feature may be only a proxy. section 9.6.2 section 9.9
Figure 9.3 Permutation importance using out-of-bag observations. The forest first establishes baseline predictions and error using observations that were omitted from the bootstrap samples of the trees evaluating them. The values of feature X are then randomly shuffled among those observations, disrupting the relationships carried by that feature while leaving the other feature columns unchanged. Without retraining the forest, the same trees recalculate their predictions from the modified data. The increase from baseline error to permuted error measures how much predictive performance depended on feature X. In Breiman’s original procedure, the error increase is calculated within each tree’s out-of-bag observations and then averaged across the forest.

Trace the baseline, feature shuffle, and unchanged-tree predictions used to calculate the error increase. section 9.6.1


