Evaluation assesses whether an outlier-detection system finds the kinds of unusual records that matter and ranks them appropriately. It is difficult when no ground truth, labels, or internal reference exists. A focused detection task with labels permits measuring false positives and false negatives, while unknown statistical outliers remain harder to judge. section 4.12 section 8.5
Evaluate detectors by combining available evidence rather than treating any single signal as decisive:
- Use labeled records to estimate false positives and false negatives when the target outlier type is specific enough to label. section 4.12
- Use visualization and correlations between detector scores to understand agreement, while recognizing that neither gives a complete accuracy estimate or guarantees future performance. section 8.5 section 4.12
- Generate synthetic data with deliberately inserted outliers to test coverage, ranking, and expected performance on future data, including multi-feature cases that visualization cannot expose. section 8.5
- Iterate between model and parameter choices, preprocessing, feature engineering, evaluation, and further data collection until results serve the task. section 4.1
Scores from multiple detectors
On a baseball dataset, KNN, Radius, and KDE detectors show some agreement, especially KNN and KDE, but their scores do not match perfectly. Agreement can inform interpretation without establishing that every flagged record is meaningful. section 4.12
The myth of average
A cockpit designed for the average pilot illustrates why unusualness depends on the dimensions examined: pilots could be average in most measurements without any one being average across all measurements. In high-dimensional data, many records may therefore receive non-zero outlier scores. section 4.12
Figure 4.1 Workflow outlining the basic steps in an outlier detection project. This assumes a one-time analysis (as opposed to ongoing monitoring) and skips tests for collective outliers and time-series analysis, which may also often be done. Note the loop between tuning and evaluating the model, which may continue for any number of iterations. Additional loops may also be present in some projects, as any of the steps may be repeated as necessary.
Use the workflow to locate evaluation inside the iterative cycle of fitting models, tuning parameters, comparing alternatives, following up on outliers, and improving future runs. section 4.1
