Coverage error evaluates multilabel rankings by measuring how many top-scored labels must be included before all true labels are covered. Lower values indicate better rankings; the best achievable value is the average number of true labels. section 5.4.2
Use coverage error to interpret multilabel predictions alongside other ranking and classification metrics:
- Penalize rankings that place true labels below many incorrect labels. section 5.4.2
- Compare it with ranking loss, which measures incorrectly ordered label pairs. section 5.4.2
- Consider label calibration because differing probability calibration can affect ranking quality. section 5.4.2
CYP inhibitor prediction
For a multilabel model predicting inhibition across CYP2C19, CYP2D6, CYP3A4, CYP1A2, and CYP2C9, calculate coverage error from the model's label scores to measure how far the ranking must be extended to include every true inhibition label. section 5.4.2
Listing 5.10 Create Multilabel CYP Inhibitor Data and Train and Evaluate Models
from rdkit.Chem import MolToInchi, MolFromInchi
from skmultilearn.model_selection import IterativeStratification
from sklearn.multioutput import MultiOutputClassifier
from sklearn.metrics import coverage_error, label_ranking_loss
def process_cyp_data(cyp):
df = ADME(name = f'{cyp}_Veith').get_data()
df = df.rename(columns={'Y':f'{cyp}'})
PandasTools.AddMoleculeColumnToFrame(df, "Drug", "Molecule")
df['Inchi'] = df.Molecule.apply(MolToInchi)
df = df.dropna()
return df
cyps = ['CYP2C19', 'CYP2D6', 'CYP3A4', 'CYP1A2', 'CYP2C9']
cyp_inhibitor_dfs = [process_cyp_data(cyp) for cyp in cyps]
cyp_inhibitor_df = cyp_inhibitor_dfs[0]
for df in cyp_inhibitor_dfs[1:]:
cyp_inhibitor_df = cyp_inhibitor_df.merge(df, on='Inchi', how='inner')
cyp_inhibitor_df = cyp_inhibitor_df[
['Molecule', 'CYP2C19', 'CYP2D6', 'CYP3A4', 'CYP1A2', 'CYP2C9']
]
cyp_inhibitor_df['fingerprints'] = cyp_inhibitor_df['Molecule'].apply(
compute_fingerprint, r=2, nBits=2048
)
X = cyp_inhibitor_df["fingerprints"]
y = cyp_inhibitor_df[cyps]
stratifier = IterativeStratification(
n_splits=2, order=2, sample_distribution_per_fold=[0.2, 0.8]
)
train_indexes, test_indexes = next(stratifier.split(X, y))
X_train, y_train = X.loc[train_indexes], y.loc[train_indexes]
X_test, y_test = X.loc[test_indexes], y.loc[test_indexes]
lr_clf = MultiOutputClassifier(LogisticRegression()).fit(
np.array(X_train.values.tolist()), y_train
)
lr_y_pred = lr_clf.predict(np.array(X_test.values.tolist()))
lr_y_pred_proba = lr_clf.predict_proba(np.array(X_test.values.tolist()))
rf_clf = RandomForestClassifier(n_estimators=500, n_jobs=-1).fit(
np.array(X_train.values.tolist()), y_train
)
rf_y_pred = rf_clf.predict(np.array(X_test.values.tolist()))
rf_y_pred_proba = rf_clf.predict_proba(np.array(X_test.values.tolist()))
print(f"LR Coverage Error: {coverage_error(y_test, lr_y_pred):.3f}")
# Output -> LR Coverage Error: 2.863
print(f"RF Coverage Error: {coverage_error(y_test, rf_y_pred):.3f}")
# Output -> RF Coverage Error: 2.862
print(f"LR Ranking Loss: {label_ranking_loss(y_test, lr_y_pred):.3f}")
# Output -> LR Ranking Loss: 0.481
print(f"RF Ranking Loss: {label_ranking_loss(y_test, rf_y_pred):.3f}")
# Output -> RF Ranking Loss: 0.476
The listing shows a Scikit-learn evaluation workflow that imports coverage_error for a multilabel CYP inhibitor task. section 5.4.2
