Permutation feature importance
Permutation feature importance measures a feature’s importance to a model by randomly permuting its values and assessing the resulting change in predictive performance.
Ensemble Methods for Machine Learning
Permutation feature importance estimates a feature's contribution by measuring the change in model performance after that feature's values are randomly permuted.
- How it is used
- Supports global, model-agnostic interpretation when model internals are unavailable.
- How deep it goes
- Explains the shuffle-and-reevaluate procedure, test-set requirement, metric choice, and repeated computation across features.
- Examples
- Uses an XGBoost model evaluated with balanced accuracy or recall and 30 permutation repeats.
- What you can do after
- Apply it via scikit-learn to rank features by their effect on hold-out predictive performance.
Read how Ensemble Methods for Machine Learning explains permutation feature importance →
Timeless Algorithms, The Foundational Ideas
Permutation importance measures a feature's contribution to a model by assessing the effect of permuting its values on predictive performance.
- How it is used
- Quantifies how much a trained random forest’s out-of-bag predictive performance depends on each feature.
- How deep it goes
- Defines baseline and permuted error, feature shuffling, and forest averaging; cautions that the score does not establish causality.
- Examples
- Contrasts Celsius and Fahrenheit predictors that encode the same temperature information.
- What you can do after
- Evaluate a feature’s predictive reliance by comparing out-of-bag error before and after shuffling its values.
Read how Timeless Algorithms, The Foundational Ideas explains permutation feature importance →
Machine Learning System Design
Permutation feature importance measures a feature's importance by randomly shuffling its values and measuring the resulting change in model performance.
- How it is used
- Supports model-agnostic feature selection and interpretation by measuring each feature’s contribution to model performance.
- How deep it goes
- Defines random permutation and performance-decrease measurement without detailing implementation procedures or statistical caveats.
- Examples
- Uses shuffle importance in retail forecasting alongside SHAP, LIME, coefficients, and gradient-boosting split counts.
- What you can do after
- Apply it to compare feature contributions and identify features that contribute little to model outcomes.
Read how Machine Learning System Design explains permutation feature importance →
Also covered in
What to know first
- decision treeA decision tree is a tree-structured model or representation in which successive tests or choices lead through branches to outcomes, predictions, or actions.
- gradient boostingGradient boosting is an ensemble-learning technique that sequentially adds predictive models fitted to reduce a loss by following its gradient.
- linear regressionLinear regression is a statistical method that models a numeric response as a linear function of one or more predictor variables with fitted parameters.
- random forestA random forest is an ensemble learning model that aggregates predictions from multiple decision trees trained with randomized data samples, feature subsets, or both.
Related concepts
- feature importance scorekind of · A feature importance score is a numerical estimate of an input feature’s relative contribution or influence on a model’s predictions or performance.
- independent variablemeasures · An independent variable is an input or explanatory variable used to model, explain, or predict variation in an outcome, without necessarily causing it.
- model-agnostic methodkind of · A model-agnostic method analyzes predictions or estimates explanations without relying on a model’s internal structure, implementation, or architecture.
- out-of-bag erroruses · Out-of-bag error is an estimate of an ensemble’s prediction error computed for each observation using only ensemble members whose bootstrap samples omitted it.
- test errormeasures · Test error is the frequency or magnitude of a model’s prediction errors on data held out from fitting for evaluation.
- evaluation datasetuses · An evaluation dataset is a collection of data or test cases used to assess a model or system against specified outcomes or criteria.
Questions about permutation feature importance
What is permutation feature importance?
Permutation feature importance measures a feature’s importance to a model by randomly permuting its values and assessing the resulting change in predictive performance.
Which Manning books explain permutation feature importance?
Ensemble Methods for Machine Learning, Timeless Algorithms, The Foundational Ideas and Machine Learning System Design.
What should I know before learning permutation feature importance?
Manning books that teach permutation feature importance usually introduce decision tree, gradient boosting, linear regression and random forest first.
What is permutation feature importance a kind of?
Permutation feature importance is a kind of feature importance score. A feature importance score is a numerical estimate of an input feature’s relative contribution or influence on a model’s predictions or performance.
What is permutation feature importance also called?
Permutation feature importance is also called permutation importance.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


