Importance score

As explained in Ensemble Methods for Machine Learning

Feature importance is a score representing the contribution or relevance of an input feature to a model's predictions.

An importance score ranks an input feature by its contribution to a model’s decisions. It supports feature selection and global interpretability: low-scoring features may be removable with little loss in model quality, while reducing training and prediction cost. Scores are commonly normalized to sum to 1, so they act as relative feature weights rather than standalone measures of causal effect. section 2.3.3 section 9.6

For a decision tree, importance is based on how much a feature improves splits across the tree. Each split’s entropy decrease is weighted by the examples reaching that node, then contributions are accumulated by feature and normalized. A feature never used in a split receives zero importance. section 9.1.2

Tree ensembles extend this idea by aggregating their trees’ scores:

• A random forest reports feature importances through feature_importances_, allowing features to be ranked and filtered with an inspected or validation-selected threshold. section 2.5.3
• An ensemble importance can be computed by averaging importances from its base decision trees; scikit-learn random forests and XGBoost provide these aggregate scores. section 9.2.3
• Interpret rankings carefully when features are strongly correlated: interchangeable inputs can carry similar information, while their order of use can affect tree-based scores. section 2.3.3

Permutation feature importance offers a model-agnostic alternative. Start with performance on a hold-out test set, shuffle one feature’s values, and measure the resulting increase in test error. A large degradation indicates that the model depended on that feature; little change indicates limited contribution. Repeat for every feature and use repeated permutations for a more robust estimate. section 9.3.1

Figure 9.7 The procedure for computing permutation feature importance illustrated for the third feature. This procedure is repeated for all features. Permutation feature importance uses only inputs and outputs to estimate feature importance and doesn’t use model internals (making this a model-agnostic approach).

The procedure visualizes the before-and-after test evaluation for one shuffled feature, then repeats it across all features without inspecting model internals. section 9.3.1

Breast cancer feature selection

A random forest can rank diagnostic measurements for benign-versus-malignant classification. With an importance threshold of 0.02, the selected features include mean concave points, worst area, and worst concave points. Comparing this ranking with label correlations can be informative, but correlation alone may miss nonlinear feature-label relationships. section 2.5.3

Importance methods answer different questions. Split-based scores expose how tree models used features internally; permutation scores test how disrupting each feature changes predictive performance. Both are global explanations, describing overall model behavior rather than explaining one individual prediction. section 9.2.3 section 9.3.1

How Ensemble Methods for Machine Learning covers it

How deep it goes

Derives weighted entropy-decrease scores for trees, then contrasts ensemble aggregation with hold-out permutation testing.

Examples

Uses two-dimensional data, Iris tree splits, breast cancer diagnosis, and XGBoost ensemble comparisons.

What it assumes

Assumes familiarity with supervised prediction, test sets, decision-tree splits, and evaluation metrics while explaining score calculations.

What you can do after

Evaluate/understand feature rankings from tree ensembles and test feature reliance through hold-out permutation scores.

How it connects

Connects decision trees, random forests, XGBoost, entropy, feature selection, correlation, and permutation feature importance.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning System DesignA quantitative attribution describing how much an input feature contributes to a model's output or predictive performance.
• 

  liveBookmanning.com
  Outlier Detection in PythonFeature importance is a quantitative indication of how much an input feature contributes to a model’s prediction, score, or decision.
• 

  liveBookmanning.com
  Statistics Every Programmer NeedsFeature importance is a measure of how strongly each input feature contributes to a model's predictions, often based on its effect on split quality.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIFeature importance is a measure of how strongly an input feature contributes to a model's predictions or performance.
• 

  liveBookmanning.com
  Financial AI in PracticeFeature importance is a measure of the contribution or influence of an input feature on a model’s predictions.

Compare how 6 books explain importance score →

Related concepts

• random forestused by · A random forest is an ensemble model that combines predictions from multiple decision trees built with randomized data samples, feature selections, or both.
• permutation feature importanceincludes · Permutation feature importance measures a feature’s importance to a model by randomly permuting its values and assessing the resulting change in predictive performance.
• tree-based methoduses · A tree-based method is a predictive or decision-making technique that represents its rules or decision structure using one or more trees.
• SHAPmeasured by · SHAP is a feature-attribution method that explains model outputs by assigning input features contributions based on Shapley values.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.