AUROC measures how well a scoring model ranks positive cases above negative cases across classification thresholds. A perfect ranking scores 1.0; random ranking scores 0.5. It evaluates scores, not the quality of any single threshold or its binary labels. section 8.5.2
Use AUROC to compare rankings, while keeping its evaluation conditions in view:
- Evaluation requires ground-truth labels. In outlier detection, synthetic data made by modifying real records can provide approximate labels, but doped records may not rank above every real record. section 8.5.2 section 8.5.3
- Compare detectors and parameter choices using AUROC, then select useful decision thresholds for the intended context. A strong ranking does not by itself determine the best binary labels. section 8.5.2
- Test robustness across settings: AUROC can vary with training-set size, and extreme training values can sharply reduce some detectors’ scores. section 8.5.4 section 8.5.5
Network-event detection
On the KDD Cup dataset, Isolation Forest scores produced an AUROC of about 0.95, with repeated runs varying because the detector is stochastic. The scores were evaluated against known attack labels; the result measures ranking, not whether every flagged record is an attack. section 5.2.2
Listing 8.9 Evaluating detectors using AUROC
from sklearn.metrics import roc_auc_score test_df = pd.concat([df, doped_df]) y_true = [0]*len(df) + [1]*len(doped_df) clf = IForest() clf.fit(clean_df) y_pred = clf.decision_function(test_df) if_auroc = roc_auc_score(y_true, y_pred) clf = LOF() clf.fit(clean_df) y_pred = clf.decision_function(test_df) lof_auroc = roc_auc_score(y_true, y_pred)
The listing calculates AUROC for Isolation Forest and LOF on labeled original and doped records. section 8.5.3
