Preprocessing prepares raw data for outlier detectors by cleaning, transforming, and selecting features. It may include removing strong outliers, encoding categorical values, binning numeric fields, scaling features, filling missing values, feature engineering, and feature selection. section 4.6 section 9.0
The work is iterative rather than a fixed preliminary step: preprocessing, detector choice, parameter tuning, evaluation, feature selection, and additional data collection may be revisited until results support the task. Strong outliers may be removed before training when novelty detectors require outlier-free data, then added back for scoring. section 4.1 section 4.6
Practical decisions include:
- Fill None or NaN values because most detectors require complete inputs; median imputation is one demonstrated option. section 9.0
- Convert dates, text, phone numbers, addresses, URLs, and IP addresses into numeric or categorical representations before detection. section 9.0
- Treat cleaning as uncertain: limited checked data may poorly represent the full population, and removing records can introduce false assumptions about what is normal. section 4.6
- Vary null handling, encoding, scaling, binning, cleaning, or dimensionality reduction to create diverse detectors for an ensemble. section 14.3.2
Removing extreme numeric values
For each numeric column, calculate the first and third quartiles, derive the interquartile range, set an upper limit at Q3 + 3.5 × IQR, and retain rows beneath every limit. Median-fill a missing Strikeouts column when needed.
Figure 4.1 Workflow outlining the basic steps in an outlier detection project. This assumes a one-time analysis (as opposed to ongoing monitoring) and skips tests for collective outliers and time-series analysis, which may also often be done. Note the loop between tuning and evaluating the model, which may continue for any number of iterations. Additional loops may also be present in some projects, as any of the steps may be repeated as necessary.
Use the workflow figure to navigate the iterative relationship between preprocessing, detector selection, tuning, evaluation, and possible additional data work. section 4.1
