liveBook · Manning

Feature importance score

As explained in Build AI Drug Discovery Pipelines

Feature importance is a quantitative estimate of how much an input feature contributes to a predictive model's decisions.

A feature importance score estimates how much an input contributes to a predictive model’s decisions. In drug discovery, it can help identify which molecular descriptors or fingerprint bits influence predictions of binding or inhibition. The meaning of the score depends on how it is calculated. section 5.3.2 section 6.4.2

Random forest Gini importance measures how much a feature reduces impurity at the decision-tree nodes where it is used. The reductions are weighted by the number of training samples at each node and combined across the forest. Scores are normalized to sum to 1, so they can be used to rank features within the model. section 5.3.2

Ranking fingerprint bits

A random forest classifier predicts whether molecules inhibit CYP 3A4. Sorting its feature_importances_ values identifies the most important Morgan fingerprint bits: bit 1750 ranks first with a score of 0.01401. A bit is not as immediately interpretable as a descriptor such as molecular weight; inspecting the substructures associated with it requires additional work. section 5.3.2

Listing 5.6 Random Forest Gini Importance
bit_indices = np.argsort(rf_clf.feature_importances_)[::-1]
for i in range(10):
  print(f"{i + 1}. Fingerprint Bit {bit_indices[i]} :   {rf_clf.feature_importances_[bit_indices[i]]:.5f}")
# Output ->  1. Fingerprint Bit 1750 : 0.01401
#            2. Fingerprint Bit 1088 : 0.00986
#            3. Fingerprint Bit 1603 : 0.00905
#            4. Fingerprint Bit 1536 : 0.00899
#            5. Fingerprint Bit 875 : 0.00876
#            6. Fingerprint Bit 841 : 0.00729
#            7. Fingerprint Bit 1152 : 0.00718
#            8. Fingerprint Bit 802 : 0.00667
#            9. Fingerprint Bit 781 : 0.00630
#           10. Fingerprint Bit 1873 : 0.00623

See how to sort a fitted random forest’s importance scores and print the top ten fingerprint bits. section 5.3.2

XGBoost offers several ways to summarize feature importance across its trees:

  • Gain: the average gain of splits that use the feature. section 6.4.2
  • Coverage: the average number of samples affected by splits using the feature. section 6.4.2
  • Frequency: the number of times the feature is used to split data across the ensemble. section 6.4.2

These tree-based scores describe global model behavior; they do not by themselves explain a prediction for one molecule. For local interpretation, sensitivity analysis can estimate how a small change in a descriptor affects a single prediction. Similarity maps can also visualize which parts of a molecule influence a classifier’s prediction. section 6.4.2 section 6.4.3 section 5.3.2

Listing 6.9 Partial Derivative Analysis
def partial_derivative(model, X, feature_idx):
    delta = 5e-2
    X_plus = X.copy()
    X_plus[:, feature_idx] += delta
    y_pred_plus = model.predict_proba(X_plus)[:, 1]
    X_minus = X.copy()
    X_minus[:, feature_idx] -= delta
    y_pred_minus = model.predict_proba(X_minus)[:, 1]
    partial_derivative = (y_pred_plus - y_pred_minus) / (2 * delta)
    return partial_derivative
 
partial_derivatives = np.zeros(X_all_2.shape[1])
for i in range(X_all_2.shape[1]):
    partial_derivatives[i] = partial_derivative(
        best_model, X_all_2[:1], i)[0]
feature_names = df_all_2.drop(
    columns=['Library']
).columns[partial_derivatives > 1e-2]
partial_derivatives = partial_derivatives[partial_derivatives > 1e-2]
 
plt.bar(
    range(len(partial_derivatives)),
    partial_derivatives,
    tick_label=feature_names
)

Explore a finite-difference implementation that estimates descriptor sensitivity for one RNA binder. section 6.4.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains weighted impurity reduction and three XGBoost measures, then demonstrates ranking and local sensitivity analysis.
Examples
Ranks Morgan fingerprint bits for CYP 3A4 inhibition and examines descriptors for RNA-binding predictions.
What you can do after
Apply it via scikit-learn and XGBoost to rank features, and use sensitivity analysis to inspect individual predictions.
How it connects
Connects feature importance to random forests, gradient boosting, molecular fingerprints, sensitivity analysis, and similarity maps.

Also explained in

Compare how 10 books explain feature importance score →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.