AUC-PR, or area under the precision-recall curve, summarizes a classifier’s precision–recall balance across decision thresholds. Lowering a molecule’s classification threshold generally increases recall while reducing precision, so the curve shows how that trade-off changes. section 5.1.3
Interpret AUC-PR against the appropriate baseline:
- Higher AUC-PR is better; a perfect classifier scores 1. section 5.1.3
- A random classifier scores its positive-class proportion, also called class skew, rather than a universal 0.5. section 5.1.3
- Use the curve to judge whether a model identifies many true inhibitors without producing too many false positives. section 5.1.3
RNA-binding small molecules
A gradient-boosted classifier distinguishes 2,003 RNA-binding compounds from 77,678 protein-binding compounds using Mordred molecular descriptors. Ten-fold stratified cross-validation evaluates predicted probabilities with average precision, an AUC-PR measure. The untuned XGBClassifier reaches an average AUPRC of 0.79, compared with 0.37 for LASSO and 0.78 for a neural network. section 6.4.1
Listing 6.8 Training and Evaluating an XGBClassifier
from sklearn.metrics import average_precision_score
from sklearn.model_selection import StratifiedKFold
models = [
XGBClassifier(),
#LGBMClassifier(), #A
#CatBoostClassifier(silent=True), #A
]
best_model = None
best_score = -np.inf
best_model_name = None
for model in models:
cv = StratifiedKFold(10, random_state=5, shuffle=True)
scores = []
for idx, (train, test) in enumerate(cv.split(X_all_2, y_all_2)):
X_train, X_test = X_all_2[train], X_all_2[test]
y_train, y_test = y_all_2[train], y_all_2[test]
fold_model = clone(model) #B
fold_model.fit(X_train, y_train)
y_pred = fold_model.predict_proba(X_test)[:, 1]
scores.append(average_precision_score(y_test, y_pred))
score = sum(scores) / len(scores)
if score > best_score:
best_score = score
best_model = model
best_model_name = model.__class__.__name__
print(f"Best model: {best_model_name}, Best AUPRC: {best_score}")
# Output -> Best model: XGBClassifier, Best AUPRC: 0.7905130730469385The implementation trains XGBClassifier models across stratified folds and scores each fold with sklearn’s average_precision_score. section 6.4.1
