liveBook · Manning

Class imbalance

As explained in Build AI Drug Discovery Pipelines

Class imbalance is a dataset condition in which some outcome values or classes occur much more frequently than others.

Class imbalance occurs when some outcome classes appear much more often than others. In a dataset for predicting CYP3A4 inhibition, inhibitors are the minority at about 41% of molecules. A classifier that minimizes overall error may favor the more common noninhibitors and miss patterns that identify inhibitors. The smaller class also provides less signal, making learning more sensitive to noise and outliers. section 5.2.2 section 4.1.1

Accuracy alone can hide this problem. On the CYP3A4 validation set, a logistic regression model achieves 0.78 accuracy, but a dummy model that always predicts the negative class achieves 0.55. Precision and recall give a more nuanced view: the logistic model scores 0.76 and 0.74, respectively. Compare against a simple baseline and examine metrics that reflect performance on the class of interest. section 5.1.2 section 5.2.2

Training and evaluation offer several ways to address imbalance:

  • Undersample the majority class by removing randomly selected examples. This balances the training data but can discard useful information, especially in a small dataset. section 5.2.2
  • Oversample the minority class by duplicating randomly selected examples. Repeated examples can lead to overfitting. section 5.2.2
  • Consider probability calibration or prediction-threshold tuning alongside precision and recall when assessing predictions. section 5.2.2
Listing 5.4 Comparing Class Imbalance Mitigation Strategies
from sklearn import metrics
def pr_auc_score(clf_, X, y):
  clf_proba = clf_.predict_proba(X)[:,1]
  clf_precision, clf_recall, _ = metrics.precision_recall_curve(
    y, clf_proba
  )
  return metrics.auc(clf_recall, clf_precision)
inhibitors = train_df[train_df.Inhibitor == 1]
noninhibitors = train_df[train_df.Inhibitor == 0]
train_df_downsampled = pd.concat(
  [inhibitors, noninhibitors.sample(len(inhibitors))]
)
train_df_upsampled = pd.concat(
  [inhibitors.sample(len(noninhibitors), replace=True), noninhibitors]
)
 
tree_clf = DecisionTreeClassifier().fit(
  train_fingerprints, train_df.Inhibitor
)
down_sampled_tree_clf = DecisionTreeClassifier().fit(
  np.array(train_df_downsampled["fingerprints"].values.tolist()), 
  train_df_downsampled.Inhibitor
)
up_sampled_tree_clf = DecisionTreeClassifier().fit(
  np.array(train_df_upsampled["fingerprints"].values.tolist()), 
  train_df_upsampled.Inhibitor
)
wt_tree_clf = DecisionTreeClassifier(class_weight='balanced').fit(
  train_fingerprints, train_df.Inhibitor
)
 
print(f"LR Validation PR AUC: "
      f"{pr_auc_score(lr_model, val_fingerprints, val_df.Inhibitor):.3f}")
# Output -> LR Validation PR AUC: 0.813
print(f"DT Validation PR AUC: "
      f"{pr_auc_score(tree_clf, val_fingerprints, val_df.Inhibitor):.3f}")
# Output -> DT Validation PR AUC: 0.724
print(f"DS DT Validation PR AUC: "   # downsampled
      f"{pr_auc_score(down_sampled_tree_clf, val_fingerprints, val_df.Inhibitor):.3f}")
# Output -> DS DT Validation PR AUC: 0.721
print(f"US DT Validation PR AUC: "   # upsampled
      f"{pr_auc_score(up_sampled_tree_clf, val_fingerprints, val_df.Inhibitor):.3f}")
# Output -> US DT Validation PR AUC: 0.719
print(f"Weighted DT Validation PR AUC: "
      f"{pr_auc_score(wt_tree_clf, val_fingerprints, val_df.Inhibitor):.3f}")
# Output -> Weighted DT Validation PR AUC: 0.736

Compare decision trees trained on the original, downsampled, and upsampled CYP3A4 training data, using a precision-recall area calculation. section 5.2.2

Imbalance in an affinity regression dataset

Imbalance can also be embedded in regression outcomes. In the DAVIS drug-target affinity dataset, an assay detection limit aggregates weak and nonbinding pairs at a value of 5. A model could obtain deceptively low mean squared error by predicting 5 for most pairs while missing rarer potent binders. The Concordance Index instead assesses whether pairs are ranked in the correct affinity order. section 11.1.1

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains majority-class bias and sampling trade-offs, then compares decision-tree training strategies and evaluation metrics.
Examples
Uses CYP3A4 inhibitor classification and the DAVIS drug-target affinity distribution to illustrate imbalanced outcomes.
What you can do after
Evaluate models using baselines and imbalance-sensitive metrics, and compare undersampling and oversampling strategies.
How it connects
Connects class imbalance to precision, recall, probability calibration, threshold tuning, and the Concordance Index.

Also explained in

Compare how 13 books explain class imbalance →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.