liveBook · Manning

AUC-PR

As explained in Build AI Drug Discovery Pipelines

The area under the precision-recall curve is a summary metric that aggregates precision and recall across classification decision thresholds.

AUC-PR, or area under the precision-recall curve, summarizes a classifier’s precision–recall balance across decision thresholds. Lowering a molecule’s classification threshold generally increases recall while reducing precision, so the curve shows how that trade-off changes. section 5.1.3

Interpret AUC-PR against the appropriate baseline:

  • Higher AUC-PR is better; a perfect classifier scores 1. section 5.1.3
  • A random classifier scores its positive-class proportion, also called class skew, rather than a universal 0.5. section 5.1.3
  • Use the curve to judge whether a model identifies many true inhibitors without producing too many false positives. section 5.1.3

RNA-binding small molecules

A gradient-boosted classifier distinguishes 2,003 RNA-binding compounds from 77,678 protein-binding compounds using Mordred molecular descriptors. Ten-fold stratified cross-validation evaluates predicted probabilities with average precision, an AUC-PR measure. The untuned XGBClassifier reaches an average AUPRC of 0.79, compared with 0.37 for LASSO and 0.78 for a neural network. section 6.4.1

Listing 6.8 Training and Evaluating an XGBClassifier
from sklearn.metrics import average_precision_score
from sklearn.model_selection import StratifiedKFold
 
models = [
    XGBClassifier(),
    #LGBMClassifier(),    #A
    #CatBoostClassifier(silent=True),    #A
]
 
best_model = None
best_score = -np.inf
best_model_name = None
 
for model in models:
    cv = StratifiedKFold(10, random_state=5, shuffle=True)
    scores = []
    for idx, (train, test) in enumerate(cv.split(X_all_2, y_all_2)):
        X_train, X_test = X_all_2[train], X_all_2[test]
        y_train, y_test = y_all_2[train], y_all_2[test]
        fold_model = clone(model)    #B
        fold_model.fit(X_train, y_train)
        y_pred = fold_model.predict_proba(X_test)[:, 1]
        scores.append(average_precision_score(y_test, y_pred))
    score = sum(scores) / len(scores)
    if score > best_score:
        best_score = score
        best_model = model
        best_model_name = model.__class__.__name__
 
print(f"Best model: {best_model_name}, Best AUPRC: {best_score}")
# Output -> Best model: XGBClassifier, Best AUPRC: 0.7905130730469385

The implementation trains XGBClassifier models across stratified folds and scores each fold with sklearn’s average_precision_score. section 6.4.1

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains threshold effects, curve interpretation, perfect and random baselines, and cross-validated computation with average precision.
Examples
Uses CYP450 inhibitor detection and RNA-binding versus protein-binding small-molecule classification.
What you can do after
Evaluate molecular classifiers with sklearn average_precision_score and compare results against class-skew and model baselines.
How it connects
Connects precision, recall, decision thresholds, class skew, average precision, cross-validation, and ROC AUC.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.