liveBook · Manning

Coverage error

As explained in Build AI Drug Discovery Pipelines

Coverage error is a multilabel ranking metric measuring how many top-ranked labels must be included to cover all true labels for an instance.

Coverage error evaluates multilabel rankings by measuring how many top-scored labels must be included before all true labels are covered. Lower values indicate better rankings; the best achievable value is the average number of true labels. section 5.4.2

Use coverage error to interpret multilabel predictions alongside other ranking and classification metrics:

  • Penalize rankings that place true labels below many incorrect labels. section 5.4.2
  • Compare it with ranking loss, which measures incorrectly ordered label pairs. section 5.4.2
  • Consider label calibration because differing probability calibration can affect ranking quality. section 5.4.2

CYP inhibitor prediction

For a multilabel model predicting inhibition across CYP2C19, CYP2D6, CYP3A4, CYP1A2, and CYP2C9, calculate coverage error from the model's label scores to measure how far the ranking must be extended to include every true inhibition label. section 5.4.2

Listing 5.10 Create Multilabel CYP Inhibitor Data and Train and Evaluate Models
from rdkit.Chem import MolToInchi, MolFromInchi
from skmultilearn.model_selection import IterativeStratification
from sklearn.multioutput import MultiOutputClassifier
from sklearn.metrics import coverage_error, label_ranking_loss
 
def process_cyp_data(cyp):
  df = ADME(name = f'{cyp}_Veith').get_data()
  df = df.rename(columns={'Y':f'{cyp}'})
  PandasTools.AddMoleculeColumnToFrame(df, "Drug", "Molecule")
  df['Inchi'] = df.Molecule.apply(MolToInchi)
  df = df.dropna()
  return df
 
cyps = ['CYP2C19', 'CYP2D6', 'CYP3A4', 'CYP1A2', 'CYP2C9']
cyp_inhibitor_dfs = [process_cyp_data(cyp) for cyp in cyps]
cyp_inhibitor_df = cyp_inhibitor_dfs[0]
for df in cyp_inhibitor_dfs[1:]:
  cyp_inhibitor_df = cyp_inhibitor_df.merge(df, on='Inchi', how='inner')
 
cyp_inhibitor_df = cyp_inhibitor_df[
    ['Molecule', 'CYP2C19', 'CYP2D6', 'CYP3A4', 'CYP1A2', 'CYP2C9']
]
cyp_inhibitor_df['fingerprints'] = cyp_inhibitor_df['Molecule'].apply(
    compute_fingerprint, r=2, nBits=2048
)
 
X = cyp_inhibitor_df["fingerprints"]
y = cyp_inhibitor_df[cyps]
 
stratifier = IterativeStratification(
    n_splits=2, order=2, sample_distribution_per_fold=[0.2, 0.8]
)
train_indexes, test_indexes = next(stratifier.split(X, y))
X_train, y_train = X.loc[train_indexes], y.loc[train_indexes]
X_test, y_test = X.loc[test_indexes], y.loc[test_indexes]
 
lr_clf = MultiOutputClassifier(LogisticRegression()).fit(
    np.array(X_train.values.tolist()), y_train
)
lr_y_pred = lr_clf.predict(np.array(X_test.values.tolist()))
lr_y_pred_proba = lr_clf.predict_proba(np.array(X_test.values.tolist()))
 
rf_clf = RandomForestClassifier(n_estimators=500, n_jobs=-1).fit(
    np.array(X_train.values.tolist()), y_train
)
rf_y_pred = rf_clf.predict(np.array(X_test.values.tolist()))
rf_y_pred_proba = rf_clf.predict_proba(np.array(X_test.values.tolist()))
 
print(f"LR Coverage Error: {coverage_error(y_test, lr_y_pred):.3f}")
# Output -> LR Coverage Error: 2.863
print(f"RF Coverage Error: {coverage_error(y_test, rf_y_pred):.3f}")
# Output -> RF Coverage Error: 2.862
print(f"LR Ranking Loss: {label_ranking_loss(y_test, lr_y_pred):.3f}")
# Output -> LR Ranking Loss: 0.481
print(f"RF Ranking Loss: {label_ranking_loss(y_test, rf_y_pred):.3f}")
# Output -> RF Ranking Loss: 0.476
 

The listing shows a Scikit-learn evaluation workflow that imports coverage_error for a multilabel CYP inhibitor task. section 5.4.2

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Defines the ranking interpretation and optimum but does not derive the calculation or discuss implementation beyond Scikit-learn usage.
Examples
Uses multilabel cytochrome P450 inhibition predictions covering five CYP enzyme labels.
What it assumes
Assumes multilabel predictions and label rankings, while defining coverage error within the evaluation discussion.
What you can do after
Evaluate multilabel rankings with Scikit-learn's coverage_error metric and interpret lower values as better coverage.
How it connects
Connects multilabel classification with ranking loss, probability calibration, and micro- or macro-averaged classification metrics.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.