Clustering groups unlabeled items so that members of a group resemble one another more than they resemble items in other groups. Animal images, for example, might group by visible features such as color or background. Unlike classification, clustering does not require known category labels. section 1.2
For outlier detection, a cluster provides a relevant comparison group. A record outside every cluster may be a global outlier; one near a cluster’s edge may be unusual relative to its neighbors. Comparing sales with other sales can be more meaningful than comparing sales with payroll, especially when the groups have different sizes or densities. This approach depends on finding meaningful clusters, which real-world data does not always provide. section 3.2.3
Several methods turn clusters into outlier assessments:
- DBSCAN can leave records unassigned. Flagging those records identifies candidates for investigation, not automatically meaningful anomalies. section 3.2.3
- A Gaussian mixture model can identify low-probability regions; cluster-based local outlier factor considers distance from a large cluster center. section 7.8
- When clusters differ in shape, calculate each point’s Mahalanobis distance relative to its own cluster. Cluster size can also contribute to the score, making a small group more suspicious when other factors are equal. section 7.8.1
Inspecting unassigned points
Fit DBSCAN to two features from the wall-robot-navigation data, mark records with no cluster assignment, and plot them against the clustered records. The plot makes the flagged points visible without establishing why they are unusual. section 3.2.3
Preparation matters. Strong outliers can distort clustering, so one workflow removes high-scoring Isolation Forest outliers before fitting Gaussian mixture models and uses the Bayesian information criterion to choose a cluster count. Text needs a numeric representation first: word counts, TF-IDF values, or topic-related features can support clustering strings. The resulting cluster ID can itself be checked alongside other table features for unusual combinations. section 7.8.1 section 9.3.3
Listing 3.5 Using DBSCAN for outlier detection
from sklearn.cluster import DBSCAN
import pandas as pd
import numpy as np
from sklearn.datasets import fetch_openml
import matplotlib.pyplot as plt
import seaborn as sns
data = fetch_openml(
'wall-robot-navigation', version=1, parser='auto') #1
data_df = pd.DataFrame(data.data, columns=data.feature_names)
X = data_df[['V6', 'V19']]
clustering = DBSCAN().fit_predict(X) #2
data_df['DBScan Outliers'] = \
np.where(clustering < 0, 1, 0) #3
sns.scatterplot(data=data_df[data_df['DBScan Outliers']==False], x="V6",
y="V19", alpha=0.2, color='blue')
sns.scatterplot(data=data_df[data_df['DBScan Outliers']==True], x="V6",
y="V19", marker='*', s=200, color='red')
plt.show()
The Python listing shows how DBSCAN assignments become an unassigned-point flag and a two-feature scatterplot. section 3.2.3
