A univariate statistical test examines one feature at a time for rare, extreme, or otherwise unusual values. In outlier detection, it can flag records that differ statistically from most rows. A flagged record is not necessarily an error or an event of interest; relevance depends on the problem. section 4.2 section 4.7
Choose the test to match the feature and the question:
- For a numeric feature, identify extreme values using a threshold. One cleaning procedure calculates each column’s first and third quartiles, then excludes rows above the third quartile plus 3.5 times the interquartile range. section 4.6
- For an identifier, check its permitted range or look for duplicates. Such a feature may be useful for this focused check even if it adds noise to a later multivariate detector. section 4.7
- For multiple features, run simple tests column by column and aggregate their results. These tests tend to scale well and avoid the distance-calculation problems that can affect detectors with many features. section 3.3.5 section 3.3.6
Extreme values versus unusual combinations
An extreme home-runs-per-at-bat value can be checked as a single engineered feature. A record whose individual values look ordinary but whose combination is unusual may escape such a test. Multivariate detectors can therefore add findings that column-wise tests miss. section 4.12 section 6.9.1
Univariate tests can also be applied after PCA transforms the data into components without the original associations. Testing and weighting those components offers a way to adjust whether extreme values or deviations from patterns receive attention, but choosing useful weights can require trial and error. Compare flagged records with the results of other detectors rather than treating any one test as definitive. section 6.9.1 section 4.12
Listing 4.2 Removing extreme values from the data
limit_dict = {}
for col_name in df.columns:
if (df[col_name].dtype in ['int64', 'float64']):
q1 = df[col_name].quantile(0.25)
q3 = df[col_name].quantile(0.75)
iqr = q3 - q1
limit = q3 + (3.5*iqr)
limit_dict[col_name] = limit
cond = [True]*len(df)
for key, limit in limit_dict.items():
cond = cond & (df[key] <= limit)
print(len(df))
clean_df = df[cond]
print(len(clean_df))
The Python listing shows a concrete column-wise cleaning threshold and how rows exceeding any calculated upper limit are removed. section 4.6
