liveBook · Manning

F-score

As explained in Build AI Drug Discovery Pipelines

The F1 score is the harmonic mean of precision and recall for a classification system.

The F1 score combines precision and recall into one classification metric. In drug screening, it can help evaluate a model that predicts whether a compound blocks the hERG channel. A high F1 requires both high precision and high recall; the score ranges from 0 to 1. section 3.3.3

Precision is the fraction of positive predictions that are correct. Recall is the fraction of actual positives that the model finds. Predicting every compound as positive can achieve perfect recall, but may produce many false positives and low precision. section 3.3.3

F1 is the harmonic mean of precision and recall: F1 = 2 × precision × recall / (precision + recall). Using true positives (TP), false positives (FP), and false negatives (FN), the equivalent expression is F1 = 2TP / (2TP + FP + FN). section 3.3.3

Listing 3.12 5-fold cross validation with precision, recall, and F1
scoring = {
  'acc': 'accuracy',
  'prec_macro': 'precision_macro',
  'rec_macro': 'recall_macro',
  'f1_macro': 'f1_macro',
}
lin_cls_scores = cross_validate(
    lin_cls, train_fingerprints, train_labels, scoring=scoring,
    cv=5, return_train_score=True
)
lin_cls_scores_df = pd.DataFrame.from_dict(lin_cls_scores)
lin_cls_scores_df.describe().round(3)

Use scikit-learn to calculate macro-averaged F1 alongside accuracy, precision, and recall across five cross-validation folds. The listing also summarizes training and validation scores. section 3.3.3

Compare F1 with other metrics and a baseline, especially when classes are imbalanced. A large gap between training and validation scores can signal that a model does not generalize reliably to unseen data. section 3.3.3

Selecting and testing a hERG classifier

A grid search selected a pipeline using cross-validated F1. Its best configuration used L2 regularization and 2048-bit, radius-2 Morgan fingerprints, with a cross-validated F1 of 0.881. The selected model achieved 0.866 on the held-out test set. Confidence intervals, repeated splits, or external validation would support a firmer conclusion about performance. section 3.4.3 section 3.4.4

Listing 3.17 Test Set Evaluation
>>> from sklearn.metrics import f1_score, matthews_corrcoef
>>> gs_linear_sgd.best_params_
{'fingerprintfeaturizer__nBits': 2048,
 'fingerprintfeaturizer__radius': 2,
 'sgdclassifier__alpha': 0.1,
 'sgdclassifier__penalty': 'l2'}
>>> final_model = gs_linear_sgd.best_estimator_
>>> test_preds = final_model.predict(test_set.SMILES)
>>> f1_score(test_set.Class, test_preds)
0.8657718120805369
 

Calculate the held-out test score with scikit-learn’s f1_score after predicting with the selected estimator. section 3.4.4

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Defines F1 through precision and recall, gives equivalent formulas, and demonstrates cross-validation, tuning, and held-out evaluation.
Examples
Reports a hERG classifier’s cross-validated F1 of 0.881 and held-out test F1 of 0.866.
What you can do after
Apply it via scikit-learn to score predictions and select a classifier using cross-validation.
How it connects
Connects F1 to precision, recall, class imbalance, cross-validation, and hyperparameter tuning.

Also explained in

Compare how 20 books explain f-score →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.