A rare feature value occurs substantially less often than other values in the same categorical feature. Rarity alone does not establish that a value is an outlier: high-cardinality features can contain many equally uncommon values. section 2.2
Simple thresholds are fragile. Flagging values below a fixed count, selecting the N rarest values, or using a 1% frequency rule can underreport or overreport depending on the distribution and cardinality. section 2.2
A more defensible workflow is to:
- Sort category counts from most to least frequent and inspect substantial drops between adjacent values. section 2.2
- Calculate cumulative frequency for the rarest values, choose a small cumulative fraction, and derive the corresponding minimum count. section 2.2
- Treat flagged values as candidates for investigation rather than automatically meaningful anomalies. section 2.2
Categorical and email-domain cases
For the SpeedDating age_o feature, missing values are filled with the median, counts are sorted, and a cumulative fraction is used to place a cutoff. For email data, extracting the domain after '@' reduces the field to categorical values, allowing rare domains to be identified when they may indicate informative patterns such as bulk-created bogus accounts. section 2.2 section 9.2.3
Listing 2.6 Determining the count threshold for outliers using cumulative counts
import pandas as pd
import numpy as np
from sklearn.datasets import fetch_openml
import matplotlib.pyplot as plt
import seaborn as sns
data = fetch_openml('SpeedDating', version=1, parser='auto') #1
data = pd.DataFrame(data.data, columns=data.feature_names)
col_name = 'age_o'
data[col_name] = \
data[col_name].fillna(data[col_name].median()).\
astype(np.int64) #2
vc = data[col_name].value_counts() #3
cumm_frac = [vc.values[::-1][:x+1].sum() / len(data) #4
for x in range(len(vc))]
cumm_frac = np.array(cumm_frac)
num_rare_vals = np.where(cumm_frac < 0.005)[0].max() #5
cut_off = vc.values[::-1][num_rare_vals]
min_count = vc[cut_off]
plt.subplots(figsize=(10, 2))
s= sns.barplot(x=vc.index, y=vc.values, order=vc.index, color='blue')
s.axvline(len(vc) - num_rare_vals - 0.5)
s.set_title(col_name)
plt.show()
Use the listing to inspect a concrete pandas implementation of cumulative frequency thresholding. section 2.2
