Correlation measures statistical association between paired numerical quantities. In outlier detection, it compares detector scores to assess whether methods agree, especially about records ranked as outliers rather than ordinary inliers. section 8.4
Pearson correlation measures linear association between raw scores, while Spearman correlation measures Pearson correlation between score ranks. Spearman is often more useful because detectors can produce differently distributed scores with strong rank agreement but weak linear agreement. section 8.4
Apply correlation carefully when comparing outlier detectors:
- Exclude most low scores before calculating agreement; differences among records that every detector treats as inliers are usually uninformative. section 8.4 section 8.8
- Retain rows where at least one detector assigns a top-percentile score, then compare rankings or cleaned scores with Spearman correlation. section 8.4
- Use correlation to seek diversity in ensembles, but balance low correlation against accuracy because a weak detector may appear diverse simply because it performs poorly. section 14.4.5 section 14.3.1
Comparing detector outputs
On the abalone dataset, several PyOD detectors produce scores that can be ranked and filtered to their highest values before creating a Spearman-correlation heatmap. Many detectors may correlate strongly without agreeing perfectly; LOF and LDOF can differ noticeably from the others.
