Pseudo-ground truth is a set of approximate inlier and outlier labels reliable enough to evaluate detectors or construct an ensemble, even though some labels may be wrong. It is useful when sufficient labeled data or reliable synthetic data is unavailable. Its value depends on how the labels were established; agreement with them is not proof that every real outlier has been found. section 14.4 section 4.12
Two ways to create it trade manual effort for confidence:
- Inspect a superset: run many diverse detectors, varying preprocessing and parameters; collect every record flagged by at least one; remove duplicates; and manually decide which are relevant outliers. Check records just below the cutoffs too. This aims to capture nearly all detectable outliers of interest, but may produce too many candidates to inspect. section 14.4.1
- Use detector consensus: scale and combine scores from many detectors to produce an approximate target without manually inspecting every flagged record. This is more practical, but rests on the assumption that most contributing detectors work reasonably well and may be a weaker approximation. section 14.4.2
Listing 14.6 Collecting a superset of high-scored records
def get_top_scored(df, scores_col):
cutoff = sorted(df[scores_col])[-10]
return scores_df[scores_df[scores_col] >= cutoff].index.tolist()
scores_cols = set(scores_df.columns) - set(df.columns) #1
top_rows = []
for scores_col in scores_cols:
res = get_top_scored(scores_df, scores_col) #2
top_rows.extend(res)
top_rows = list(set(top_rows)) #3
The Python listing collects the highest-scored rows from each detector and deduplicates their indices. Its top-ten cutoff illustrates a way to form an inspection set, not a generally valid threshold. section 14.4.1
Once established, the target can guide selection of a smaller ensemble. Compare candidate detector subsets with the target while considering training time, prediction time, interpretability, and diversity. A greedy search can add the detector that best improves agreement at each step, or remove detectors from a full set. Agreement need not improve with every addition, and matching the target exactly is unlikely. section 14.4.2 section 14.4.4
More detectors need not mean a better match
In one greedy selection run, agreement with the target rises from 0.88 with one detector to 0.89 with two, then falls to 0.87 with three. The extra detector should be judged against costs and other goals rather than retained automatically. section 14.4.4
Keep the target's limitations visible. Inspecting only low-scored records is unlikely to reveal many missed outliers because outliers are rare; a detector superset can still miss records, while consensus can reinforce shared mistakes. Low correlation alone is no guarantee of accuracy: a poor detector may simply disagree with the others. Where possible, compare against labeled or synthetic data, and choose evaluation metrics suited to the task. section 4.12 section 14.4.5 section 14.2
