A global explanation describes how an outlier detector behaves across the dataset: the patterns it discovers, how those patterns produce scores, and which properties are associated with the highest scores. It differs from a local explanation, which accounts for one record's score. section 13.1.2
Global explanations can support several practical tasks:
- Assess whether the detector emphasizes relevant patterns or overlooks important anomalies. section 13.1.2
- Debug false emphasis and missed anomalies when the explanation is clear and accurate. section 13.1.2
- Compare detectors when forming ensembles by identifying which types of outliers each method detects. section 13.1.2
- Summarize distance distributions and influential features for distance- or density-based detectors, or summarize isolation trees for Isolation Forest. section 13.1.2
Aggregating explanations from two-dimensional models
An ensemble of two-dimensional KNN models can identify feature pairs that contribute most to outlier scores. Aggregating those pairs across the data produces a global explanation listing features most associated with detected outliers. In an abalone example, Height appears in the strongest subspaces, indicating that it is repeatedly associated with anomalous records. section 13.3.1
Listing 13.10 Calculating outlier scores with a set of 2D KNN models
import pandas as pd
import numpy as np
from sklearn.datasets import fetch_openml
from sklearn.preprocessing import RobustScaler, OrdinalEncoder
from pyod.models.knn import KNN
import matplotlib.pyplot as plt
import seaborn as sns
np.random.seed(0)
data = fetch_openml('abalone', version=1, parser='auto') #1
df = pd.DataFrame(data.data, columns=data.feature_names)
df = pd.DataFrame(OrdinalEncoder().fit_transform(df), columns=df.columns)
df = pd.DataFrame(RobustScaler().fit_transform(df), columns=df.columns)
orig_features = list(df.columns)
det = KNN() #2
det.fit(df)
df['KNN Score'] = det.decision_function(df)
combo_id = 0 #3
scores_df = pd.DataFrame()
scores_2d_cols = []
for col_name_1_idx, col_name_1 in enumerate(orig_features):
for col_name_2_idx, col_name_2 in enumerate(
orig_features[col_name_1_idx+1:]):
print(col_name_1, col_name_2)
det = KNN()
det.fit(df[[col_name_1, col_name_2]])
scores_df[str(combo_id)] = det.decision_function(
df[[col_name_1, col_name_2]])
scores_2d_cols.append((col_name_1, col_name_2))
combo_id += 1
scores_df['Final Score'] = scores_df.sum(axis=1) #4
scores_df = scores_df.sort_values('Final Score')
df['KNN 2D Scores'] = scores_df['Final Score']
sns.scatterplot(data=df, x='KNN Score', y='KNN 2D Scores') #5
plt.show()
df = df.sort_values(['KNN 2D Scores']) #6
max_row = df.iloc[-1:]
top_row_index = max_row.index[0]
top_feature_pairs_idxs = np.argsort(
scores_df.loc[top_row_index])[::-1][1:5].values
fig, ax = plt.subplots(nrows=2, ncols=2, figsize=(8, 8)) #7
for pair_idx, pair_id in enumerate(top_feature_pairs_idxs):
col_pairs = scores_2d_cols[pair_id]
s = sns.scatterplot(data=df, x=col_pairs[0], y=col_pairs[1], alpha=0.3,
ax=ax[pair_idx//2][pair_idx%2])
s = sns.scatterplot(data=max_row, x=col_pairs[0], y=col_pairs[1],
ax=ax[pair_idx//2][pair_idx%2], color='red',
marker='*', s=200)
s.set_title(f"{col_pairs[0]} \nAND \n{col_pairs[1]}")
plt.tight_layout()
plt.show()
A concise Python workflow demonstrates preprocessing the abalone data, fitting KNN models across feature pairs, and calculating the resulting outlier scores. section 13.3.1
