A numeric detector is an outlier-detection model that analyzes numerical features or numeric data. Because most detectors expect numeric inputs, categorical columns usually require one-hot or ordinal encoding; distance-based detectors generally also require scaled numeric features. section 3.3.2 section 5.1
Practical use typically involves:
- Preparing representative data by handling null values, engineering or selecting features, encoding categorical columns, and scaling numeric features where required. section 4.15 section 4.6
- Removing a small number of especially strong outliers before fitting detectors when they could distort the model of normal data; scored records may later be added back. section 4.6
- Comparing detectors rather than treating any flagged record as automatically meaningful, since evaluation is difficult and often requires manual inspection. section 4.15
- Using scikit-learn’s Isolation Forest, LocalOutlierFactor, or KNN as common starting points; the available detectors return scores and labels, with −1 for outliers and 1 for inliers. section 5.1
- Using PyOD when a broader, consistent collection of numeric-tabular detectors is useful, while recognizing that interpretability and parameter support vary by detector. section 6.15
Removing extreme numeric values before detection
A preprocessing pass can calculate each numeric column’s first and third quartiles, retain values below the upper limit q3 + 3.5 × IQR, and fit the detector on the resulting data. This is a cleaning heuristic, not proof that removed records are anomalous. section 4.6
Listing 4.2 Removing extreme values from the data
limit_dict = {}
for col_name in df.columns:
if (df[col_name].dtype in ['int64', 'float64']):
q1 = df[col_name].quantile(0.25)
q3 = df[col_name].quantile(0.75)
iqr = q3 - q1
limit = q3 + (3.5*iqr)
limit_dict[col_name] = limit
cond = [True]*len(df)
for key, limit in limit_dict.items():
cond = cond & (df[key] <= limit)
print(len(df))
clean_df = df[cond]
print(len(clean_df))
Inspect the complete listing for an IQR-based procedure that builds numeric-column limits and filters extreme records. section 4.6
