A feature importance score estimates how much an input contributes to a predictive model’s decisions. In drug discovery, it can help identify which molecular descriptors or fingerprint bits influence predictions of binding or inhibition. The meaning of the score depends on how it is calculated. section 5.3.2 section 6.4.2
Random forest Gini importance measures how much a feature reduces impurity at the decision-tree nodes where it is used. The reductions are weighted by the number of training samples at each node and combined across the forest. Scores are normalized to sum to 1, so they can be used to rank features within the model. section 5.3.2
Ranking fingerprint bits
A random forest classifier predicts whether molecules inhibit CYP 3A4. Sorting its feature_importances_ values identifies the most important Morgan fingerprint bits: bit 1750 ranks first with a score of 0.01401. A bit is not as immediately interpretable as a descriptor such as molecular weight; inspecting the substructures associated with it requires additional work. section 5.3.2
Listing 5.6 Random Forest Gini Importance
bit_indices = np.argsort(rf_clf.feature_importances_)[::-1]
for i in range(10):
print(f"{i + 1}. Fingerprint Bit {bit_indices[i]} : {rf_clf.feature_importances_[bit_indices[i]]:.5f}")
# Output -> 1. Fingerprint Bit 1750 : 0.01401
# 2. Fingerprint Bit 1088 : 0.00986
# 3. Fingerprint Bit 1603 : 0.00905
# 4. Fingerprint Bit 1536 : 0.00899
# 5. Fingerprint Bit 875 : 0.00876
# 6. Fingerprint Bit 841 : 0.00729
# 7. Fingerprint Bit 1152 : 0.00718
# 8. Fingerprint Bit 802 : 0.00667
# 9. Fingerprint Bit 781 : 0.00630
# 10. Fingerprint Bit 1873 : 0.00623See how to sort a fitted random forest’s importance scores and print the top ten fingerprint bits. section 5.3.2
XGBoost offers several ways to summarize feature importance across its trees:
- Gain: the average gain of splits that use the feature. section 6.4.2
- Coverage: the average number of samples affected by splits using the feature. section 6.4.2
- Frequency: the number of times the feature is used to split data across the ensemble. section 6.4.2
These tree-based scores describe global model behavior; they do not by themselves explain a prediction for one molecule. For local interpretation, sensitivity analysis can estimate how a small change in a descriptor affects a single prediction. Similarity maps can also visualize which parts of a molecule influence a classifier’s prediction. section 6.4.2 section 6.4.3 section 5.3.2
Listing 6.9 Partial Derivative Analysis
def partial_derivative(model, X, feature_idx):
delta = 5e-2
X_plus = X.copy()
X_plus[:, feature_idx] += delta
y_pred_plus = model.predict_proba(X_plus)[:, 1]
X_minus = X.copy()
X_minus[:, feature_idx] -= delta
y_pred_minus = model.predict_proba(X_minus)[:, 1]
partial_derivative = (y_pred_plus - y_pred_minus) / (2 * delta)
return partial_derivative
partial_derivatives = np.zeros(X_all_2.shape[1])
for i in range(X_all_2.shape[1]):
partial_derivatives[i] = partial_derivative(
best_model, X_all_2[:1], i)[0]
feature_names = df_all_2.drop(
columns=['Library']
).columns[partial_derivatives > 1e-2]
partial_derivatives = partial_derivatives[partial_derivatives > 1e-2]
plt.bar(
range(len(partial_derivatives)),
partial_derivatives,
tick_label=feature_names
)Explore a finite-difference implementation that estimates descriptor sensitivity for one RNA binder. section 6.4.3









