Cluster-based outlier detection looks for records that do not fit the cluster structure of a dataset. The starting idea is that ordinary records lie near the centers of large, dense clusters, while unusual records lie farther away. This is most useful when the data forms meaningful clusters; otherwise, the comparison can be misleading. section 3.2.3
Clustering supports two kinds of comparison:
- Global outliers are not assigned to any cluster: too few similar records surround them to form one. section 3.2.3
- Local outliers sit near a cluster’s edge. They resemble some records but remain unusual relative to that cluster. section 3.2.3
For tabular distance-based clustering, encode categorical features numerically and put numeric features on comparable scales, typically with min-max scaling or standardization. Otherwise, large-scale features can dominate distances. Very high-dimensional data can also make distance comparisons difficult. section 3.2
Compare like transactions
If sales, purchase, and payroll transactions form separate clusters, a sales record may be more meaningfully compared with other sales records than with every transaction. The clusters can differ in size, spacing, and dollar values, so a single dataset-wide notion of unusual may be unhelpful. section 3.2.3
DBSCAN provides a direct implementation: fit it to the selected features and flag records it assigns to no cluster. On two wall-robot-navigation features, most points form three clusters while unassigned points are marked as outliers. That flag identifies unclustered points; it does not, by itself, provide the same assessment of points at a cluster’s edge. section 3.2.3
Listing 3.5 Using DBSCAN for outlier detection
from sklearn.cluster import DBSCAN
import pandas as pd
import numpy as np
from sklearn.datasets import fetch_openml
import matplotlib.pyplot as plt
import seaborn as sns
data = fetch_openml(
'wall-robot-navigation', version=1, parser='auto') #1
data_df = pd.DataFrame(data.data, columns=data.feature_names)
X = data_df[['V6', 'V19']]
clustering = DBSCAN().fit_predict(X) #2
data_df['DBScan Outliers'] = \
np.where(clustering < 0, 1, 0) #3
sns.scatterplot(data=data_df[data_df['DBScan Outliers']==False], x="V6",
y="V19", alpha=0.2, color='blue')
sns.scatterplot(data=data_df[data_df['DBScan Outliers']==True], x="V6",
y="V19", marker='*', s=200, color='red')
plt.show()
The Python listing fits scikit-learn DBSCAN, converts negative cluster labels into outlier flags, and plots flagged versus other records. section 3.2.3
A cluster-relative density approach first clusters the records, then compares each point’s kernel density estimate (KDE) with its cluster’s average KDE. The resulting ratio expresses density relative to what is normal for that cluster and remains interpretable across different clusterings. Repeating k-means with different cluster counts and runs, then averaging results, reduces reliance on one potentially unstable partition. Even so, clustering quality matters: plausible partitions can differ, and distance-based or density-based detectors may be more reliable when the data does not cluster well. section 7.8.2 section 3.2.3
