Feature importance score
A feature importance score is a numerical estimate of an input feature’s relative contribution or influence on a model’s predictions or performance.
Rearchitecting LLMs
An importance score numerically estimates how important a model component is to its information or performance.
- How it is used
- Ranks blocks, neuron pairs, and attention modules as candidates for structured pruning.
- How deep it goes
- Works through hook-based block scoring, static and hybrid neuron metrics, safety checks, and score-based selection.
- Examples
- Compares WinoGrande and WikiText block selections and calculates neuron scores from projection weight ranges.
- What you can do after
- Apply it using PyTorch hooks and top-k selection to identify pruning candidates, then evaluate the resulting models.
Read how Rearchitecting LLMs explains feature importance score →
Machine Learning System Design
A quantitative attribution describing how much an input feature contributes to a model's output or predictive performance.
- How it is used
- Supports model transparency, deployment verification, and feature-selection decisions.
- How deep it goes
- Compares model-specific and model-agnostic methods and outlines deep-learning techniques without deriving their algorithms.
- Examples
- Illustrates importance through taxi pricing, retail demand forecasting, and explanations for search-ranking complaints.
- What you can do after
- Evaluate and understand feature contributions using model-specific and model-agnostic approaches to guide feature selection and verification.
Read how Machine Learning System Design explains feature importance score →
Build AI Drug Discovery Pipelines
Feature importance is a quantitative estimate of how much an input feature contributes to a predictive model's decisions.
- How it is used
- Uses feature importance to interpret molecular classifiers for CYP 3A4 inhibition and RNA-binding activity.
- How deep it goes
- Explains weighted impurity reduction and three XGBoost measures, then demonstrates ranking and local sensitivity analysis.
- Examples
- Ranks Morgan fingerprint bits for CYP 3A4 inhibition and examines descriptors for RNA-binding predictions.
- What you can do after
- Apply it via scikit-learn and XGBoost to rank features, and use sensitivity analysis to inspect individual predictions.
Read how Build AI Drug Discovery Pipelines explains feature importance score →
Ensemble Methods for Machine Learning
Feature importance is a score representing the contribution or relevance of an input feature to a model's predictions.
- How it is used
- Supports feature selection and communicates which inputs matter most to a model’s overall decisions.
- How deep it goes
- Derives weighted entropy-decrease scores for trees, then contrasts ensemble aggregation with hold-out permutation testing.
- Examples
- Uses two-dimensional data, Iris tree splits, breast cancer diagnosis, and XGBoost ensemble comparisons.
- What you can do after
- Evaluate/understand feature rankings from tree ensembles and test feature reliance through hold-out permutation scores.
Read how Ensemble Methods for Machine Learning explains feature importance score →
Timeless Algorithms, The Foundational Ideas
Variable importance is a quantitative assessment of how much a predictor contributes to a model's predictions or reduction of uncertainty.
- How it is used
- Uses importance scores to recover an interpretable view of how decision trees and random forests use predictors.
- How deep it goes
- Explains permutation and impurity-based importance, including out-of-bag permutation steps and limitations of interpreting scores.
- Examples
- Illustrates redundant predictors with Celsius and Fahrenheit measurements and tree importance with income and credit history in loan approval.
- What you can do after
- Evaluate and understand importance scores by distinguishing predictive dependence from split improvement and recognizing correlation and bias.
Read how Timeless Algorithms, The Foundational Ideas explains feature importance score →
Outlier Detection in Python
Feature importance is a quantitative indication of how much an input feature contributes to a model’s prediction, score, or decision.
- How it is used
- Identifies features relevant to outlier scores when a detector’s scores or labels alone do not explain flagged records.
- How deep it goes
- Distinguishes global from local importance and demonstrates plotting workflows, while emphasizing that rankings need not fully explain scores.
- Examples
- Uses abalone outlier scores, SHAP plots, PyOD IForest rankings, and BHAD’s local and global explanations.
- What you can do after
- Apply it via SHAP or supported detectors to rank relevant features, then assess whether those rankings adequately explain an outlier score.
Read how Outlier Detection in Python explains feature importance score →
Statistics Every Programmer Needs
Feature importance is a measure of how strongly each input feature contributes to a model's predictions, often based on its effect on split quality.
- How it is used
- Interprets which input variables most affect predictions in decision trees and random forests.
- How deep it goes
- Defines impurity-based scoring, explains relative values, and demonstrates forest-wide visualization without discussing alternative importance methods.
- Examples
- Uses fourth-down conversion prediction, TO_GO, PLAY_TYPE, importance scores, and a sorted bar chart.
- What you can do after
- Apply it via RandomForestClassifier and Matplotlib to rank and visualize influential features.
Read how Statistics Every Programmer Needs explains feature importance score →
Build a DeepSeek Model (From Scratch)
Expert importance is the aggregate selection weight assigned to an expert across the tokens in a batch.
- How it is used
- Quantifies each expert’s aggregate routing weight to assess balance and supply inputs to auxiliary-loss and router-probability calculations.
- How deep it goes
- Defines the batch column sum, works through numerical scores, and connects them to coefficient of variation and mean router probability.
- Examples
- Calculates scores of 1.4, 1.0, and 1.6, then contrasts equal scores from one high-probability token and four low-probability tokens.
- What you can do after
- Evaluate expert balance by aggregate routing weight while recognizing that equal importance scores do not guarantee equal token workloads.
Read how Build a DeepSeek Model (From Scratch) explains feature importance score →
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
Feature importance is a measure of how strongly an input feature contributes to a model's predictions or performance.
- How it is used
- Uses feature importance to assess model transparency and communicate which inputs most affect predictions.
- How deep it goes
- Defines importance and demonstrates XGBoost gain scores, but stops short of detailing alternative importance methods or Keras implementation.
- Examples
- Uses a house-market dataset and an Airbnb NYC price-classification model, including gain values, feature rankings, and a bar chart.
- What you can do after
- Apply it via XGBoost’s built-in API to rank features and visualize their relative importance.
Financial AI in Practice
Feature importance is a measure of the contribution or influence of an input feature on a model’s predictions.
- How it is used
- Supports portfolio-level model review for risk managers and regulators, and helps diagnose changes in financial model behavior.
- How deep it goes
- Defines split-count, gain, permutation, and mean-absolute-SHAP approaches, including key biases, limitations, and costs.
- Examples
- Uses income and age split counts, shuffled-feature error, WoE-coded credit inputs, and feature importance for market-model diagnosis.
- What you can do after
- Evaluate which features drive predictions while accounting for the chosen importance method’s limitations.
Read how Financial AI in Practice explains feature importance score →
Also covered in
- Fight Fraud with Machine LearningIn early access (MEAP)
- Vibe EngineeringIn early access (MEAP)
- Demand Forecasting Best Practices
- Knowledge Graphs and LLMs in Action
- Microsoft Azure in Action
- The Art of AI Product Development
- Graph Algorithms for Data Science
What to know first
- random forestA random forest is an ensemble learning model that aggregates predictions from multiple decision trees trained with randomized data samples, feature subsets, or both.
- gradient boostingGradient boosting is an ensemble-learning technique that sequentially adds predictive models fitted to reduce a loss by following its gradient.
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- overfittingOverfitting is a modeling condition in which a model adapts too closely to particular data, including its noise or idiosyncrasies, and consequently generalizes poorly to new data.
Related concepts
- random forestused by · A random forest is an ensemble learning model that aggregates predictions from multiple decision trees trained with randomized data samples, feature subsets, or both.
- permutation feature importanceincludes · Permutation feature importance measures a feature’s importance to a model by randomly permuting its values and assessing the resulting change in predictive performance.
- attentionmeasures · Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- causationcontrasts with · Causation is a relationship in which a change in one event, factor, or condition produces or influences a change in another.
Questions about feature importance score
What is feature importance score?
A feature importance score is a numerical estimate of an input feature’s relative contribution or influence on a model’s predictions or performance.
Which Manning books explain feature importance score?
Rearchitecting LLMs, Machine Learning System Design, Build AI Drug Discovery Pipelines, Ensemble Methods for Machine Learning, Timeless Algorithms, The Foundational Ideas, Outlier Detection in Python, Statistics Every Programmer Needs, Build a DeepSeek Model (From Scratch) and 2 more.
What should I know before learning feature importance score?
Manning books that teach feature importance score usually introduce random forest, gradient boosting, accuracy and overfitting first.
What is the difference between feature importance score and causation?
A feature importance score is a numerical estimate of an input feature’s relative contribution or influence on a model’s predictions or performance. Causation is a relationship in which a change in one event, factor, or condition produces or influences a change in another.
What kinds of feature importance score are there?
Kinds of feature importance score include permutation feature importance.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.









