A local explanation specifies why an individual record received its particular outlier score. It differs from a global explanation, which describes how the detector evaluates records overall. section 13.1.2
Post hoc local explanations approximate the detector’s actual reasoning. For an Isolation Forest, they may describe the sparse spaces containing a record, relevant features, and data distributions, but producing such explanations is difficult. section 13.1.1 section 13.1.2
An interpretable alternative builds explanations from pairs of features: multiple two-dimensional detectors identify the subspaces contributing most to an outlier’s score. These explanations can be useful, although full-feature and combined two-dimensional scores may disagree. section 13.3.1
Explaining an abalone outlier
An ensemble of two-dimensional KNN models identifies the highest-scored record. Its four strongest subspaces all include Height, suggesting that this value is incongruous with the record’s other values. section 13.3.1
Listing 13.10 Calculating outlier scores with a set of 2D KNN models
import pandas as pd
import numpy as np
from sklearn.datasets import fetch_openml
from sklearn.preprocessing import RobustScaler, OrdinalEncoder
from pyod.models.knn import KNN
import matplotlib.pyplot as plt
import seaborn as sns
np.random.seed(0)
data = fetch_openml('abalone', version=1, parser='auto') #1
df = pd.DataFrame(data.data, columns=data.feature_names)
df = pd.DataFrame(OrdinalEncoder().fit_transform(df), columns=df.columns)
df = pd.DataFrame(RobustScaler().fit_transform(df), columns=df.columns)
orig_features = list(df.columns)
det = KNN() #2
det.fit(df)
df['KNN Score'] = det.decision_function(df)
combo_id = 0 #3
scores_df = pd.DataFrame()
scores_2d_cols = []
for col_name_1_idx, col_name_1 in enumerate(orig_features):
for col_name_2_idx, col_name_2 in enumerate(
orig_features[col_name_1_idx+1:]):
print(col_name_1, col_name_2)
det = KNN()
det.fit(df[[col_name_1, col_name_2]])
scores_df[str(combo_id)] = det.decision_function(
df[[col_name_1, col_name_2]])
scores_2d_cols.append((col_name_1, col_name_2))
combo_id += 1
scores_df['Final Score'] = scores_df.sum(axis=1) #4
scores_df = scores_df.sort_values('Final Score')
df['KNN 2D Scores'] = scores_df['Final Score']
sns.scatterplot(data=df, x='KNN Score', y='KNN 2D Scores') #5
plt.show()
df = df.sort_values(['KNN 2D Scores']) #6
max_row = df.iloc[-1:]
top_row_index = max_row.index[0]
top_feature_pairs_idxs = np.argsort(
scores_df.loc[top_row_index])[::-1][1:5].values
fig, ax = plt.subplots(nrows=2, ncols=2, figsize=(8, 8)) #7
for pair_idx, pair_id in enumerate(top_feature_pairs_idxs):
col_pairs = scores_2d_cols[pair_id]
s = sns.scatterplot(data=df, x=col_pairs[0], y=col_pairs[1], alpha=0.3,
ax=ax[pair_idx//2][pair_idx%2])
s = sns.scatterplot(data=max_row, x=col_pairs[0], y=col_pairs[1],
ax=ax[pair_idx//2][pair_idx%2], color='red',
marker='*', s=200)
s.set_title(f"{col_pairs[0]} \nAND \n{col_pairs[1]}")
plt.tight_layout()
plt.show()
Shows preprocessing an abalone dataset, fitting KNN models, and calculating scores across two-dimensional feature combinations. section 13.3.1
