liveBook · Manning

Impurity-based importance

As explained in Timeless Algorithms, The Foundational Ideas

Impurity-based importance is a measure that aggregates the impurity reductions attributed to a feature across a model's tree splits.

Impurity-based importance summarizes how much a feature contributes to improved splits across a forest. Unlike permutation importance, which measures the increase in prediction error after disrupting a feature, it derives importance from the trees’ splits. section 9.6.1 section 9.9

The score has important limitations. Features with many possible split points—such as continuous variables or categorical variables with many levels—have more chances to produce favorable reductions in impurity or variance. Because the score comes from the data used to train the trees, it can also reward sample-specific patterns. It measures predictive reliance, not causality, and does not show whether a feature raises or lowers predictions. section 9.6.2 section 9.6.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines the score through accumulated split improvements and identifies bias sources, but gives no step-by-step calculation.
What you can do after
Evaluate feature reliance in a forest while recognizing that split-based scores can favor variables with many candidate splits.
How it connects
Contrasts with permutation importance and connects both measures to random forests.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.