A feature importance score quantifies a predictor’s contribution to a model’s behavior. In a single decision tree, the path from root to leaf offers a visible explanation, but a ranking must also account for how often a feature is used, how many observations its splits affect, and how much those splits improve the criterion. In a random forest, scores help interpret the collective behavior of many trees. section 9.6
Two common scores answer different questions. Permutation importance measures the increase in predictive error when a feature’s information is disrupted. Impurity-based importance accumulates the improvements associated with splits on that feature across the forest. The first concerns dependence of predictive performance on the feature; the second concerns its contribution to the forest’s split criteria. section 9.6.1 section 9.9
For permutation importance, first measure baseline out-of-bag error. Shuffle one feature’s values among the out-of-bag observations, leaving the other features unchanged, and recalculate predictions with the same trained trees. The score is permuted error minus baseline error. In Breiman’s original procedure, the error increase is calculated for each tree’s out-of-bag observations and averaged across the forest; the forest is not retrained. A large positive score indicates substantial reliance on the feature, while a score near zero indicates little measurable change in error. section 9.6.1
Figure 9.3 Permutation importance using out-of-bag observations. The forest first establishes baseline predictions and error using observations that were omitted from the bootstrap samples of the trees evaluating them. The values of feature X are then randomly shuffled among those observations, disrupting the relationships carried by that feature while leaving the other feature columns unchanged. Without retraining the forest, the same trees recalculate their predictions from the modified data. The increase from baseline error to permuted error measures how much predictive performance depended on feature X. In Breiman’s original procedure, the error increase is calculated within each tree’s out-of-bag observations and then averaged across the forest.

The diagram follows a feature from baseline out-of-bag predictions through shuffling and the resulting change in error. section 9.6.1
Impurity-based importance connects split improvement to uncertainty reduction. For example, an entropy-based tree favors splits with greater information gain. Features that repeatedly produce substantial reductions in entropy can receive higher importance scores; across a forest, their contributions are accumulated over many trees. section 5.3.1 section 5.3.2
Redundant temperature measurements
Suppose a forest receives temperature in both Celsius and Fahrenheit. If it relies on Celsius, shuffling Fahrenheit may barely affect error because Celsius remains available. Fahrenheit can then have low permutation importance even when temperature is highly predictive. Another forest may favor Fahrenheit instead. section 9.6.2
Importance is conditional on the data, model configuration, other predictors, and calculation method—not an intrinsic value of a feature. Correlated predictors can divide or shift importance, and shuffling one can create unrealistic combinations of values. Impurity-based scores can favor features with more possible splitting points. Proxy relationships can also distort interpretation, and predictive importance does not establish causality. section 9.6.2 section 9.9









