A statistical outlier is a record that is statistically unusual relative to most other rows, without requiring a predefined idea of which event matters. It differs from a specific outlier, such as fraud or a data-entry error, which is defined against an expected outcome and may not be statistically rare. section 4.2
Statistical unusualness can include both known and unknown outliers. A flagged record is therefore a candidate for investigation, not automatically a meaningful anomaly; different tools return different kinds of unusual records. section 4.2.3 section 4.2
Select the detection approach according to the goal:
- Use an outlier detector for records that are broadly statistically unusual; univariate tests, distance, density, and frequent-itemset methods are examples. section 4.2 section 4.3
- Use rules or classifiers when the target is a defined issue, although projects may combine rules, classifiers, and detectors. section 4.3
- Prepare data, evaluate and combine models, run them in production, and manually assess or spot-check collected statistical outliers. section 4.2.3 section 4.3
Missing values before detection
A temperature sensor may fail disproportionately at very low or very high temperatures. Preserve an indicator that the value was missing, separate likely data artifacts when appropriate, and replace nulls before applying most numeric or categorical detectors. section 9.1
Listing 9.1 Filling None and NaN values
import pandas as pd
import numpy as np
df = pd.DataFrame({"A": [1, 2, np.NaN, 4, 5, None],
"B": ['A', 'B', None, "B", "B", "F"]})
df_filled = df.copy()
df_filled['A Null'] = df['A'].isna()
df_filled['B Null'] = df['B'].isna()
df_filled['Num Null'] = df.isna().sum(axis=1)
df_filled['A'] = df['A'].fillna(df['A'].median())
df_filled['B'] = df['B'].fillna(df['B'].mode()[0])
print(df_filled)
Shows pandas preparation that records null indicators and fills numeric and categorical missing values before further detection. section 9.1
