Manual evaluation directly reviews individual records flagged by outlier detectors to decide which are relevant, identify categories, and find meaningful patterns. It converts imperfect detector output into labels or pseudo-ground truth rather than treating every statistically unusual record as important. section 14.4 section 15.2.1
A practical evaluation workflow is:
- Run diverse detectors with different preprocessing and parameters. section 14.4.1
- Collect records flagged by at least one detector, remove duplicates, and inspect the resulting superset. section 14.4.1
- Check records just below detector thresholds because ensembles may reveal mildly unusual records missed by individual detectors. section 14.4.1
- Label relevant outliers consistently, add useful categories, and tune the system to score them above other records. section 14.4.1 section 15.5
Inspection can reveal simple univariate explanations or more complex patterns spanning multiple features. It also exposes false positives: many statistically unusual records may be irrelevant, while newly flagged or unlabeled records require judgment before they enter evaluation data. section 15.2.1 section 15.5
Isolation Forest and error-rate patterns
After fitting an Isolation Forest to encoded KDD Cup data, sort records by score and inspect the highest-scored rows. Repeatedly high destination-host error rates can motivate an outlier category, while other features may reveal additional or more complex categories. section 15.2 section 15.2.1
Listing 14.6 Collecting a superset of high-scored records
def get_top_scored(df, scores_col):
cutoff = sorted(df[scores_col])[-10]
return scores_df[scores_df[scores_col] >= cutoff].index.tolist()
scores_cols = set(scores_df.columns) - set(df.columns) #1
top_rows = []
for scores_col in scores_cols:
res = get_top_scored(scores_df, scores_col) #2
top_rows.extend(res)
top_rows = list(set(top_rows)) #3
Shows how to collect the union of top-scored records from multiple detector columns and remove duplicates before manual review. section 14.4.1
