Spearman correlation measures association using the rank order of two sets of values: it is Pearson correlation calculated on their ranks rather than their raw values. This makes it useful when two outlier detectors order records similarly but produce scores on different distributions. Raw-score Pearson correlation can be weak even when the rankings agree. section 8.4
For detector evaluation, the question is often whether detectors identify and rank the same outliers—not whether they agree about ordinary records. Correlating every score can let numerous low-scoring records obscure that question. One approach keeps records placed in at least one detector’s top 5%, then compares scores on that subset. Another ranks each detector’s scores and sets ranks outside its top 5% to zero. The cutoff is a choice, not a universal rule. section 8.4
Listing 8.6 Cleaning the outlier scores to calculate their correlation
top_scores_df = scores_df[scores_cols].copy()
for col_name in top_scores_df.columns:
top_scores_df[col_name] = top_scores_df[col_name].rank()
top_scores_df[col_name] = top_scores_df[col_name].apply(
lambda x: x if x > (len(df) * 0.95) else 0.0)
The listing ranks each detector’s scores and zeroes ranks below the top-5% threshold before correlation. section 8.4
Spearman correlation also helps construct feature subspaces. For numeric features, calculate pairwise correlations, begin a subspace with a strongly associated pair, and add features strongly associated with those already selected. Keeping associated features together can make departures from their usual relationship visible. Spearman correlation misses nonmonotonic relationships; predictive models can capture more complex associations. section 10.1.2
Baseball feature subspace
At bats and Runs have a reported correlation of 0.94 in the baseball data. Grouping them in a feature subspace preserves a strong observed association; records that depart substantially from it may be outliers. section 10.1.2
Listing 10.3 Example code to identify subspaces based on correlations
def get_highest_corr(): #1
return np.unravel_index(
np.argmax(corr_matrix.values, axis=None),
corr_matrix.shape)
def get_correlated_subspaces(corr_matrix, num_base_detectors,
num_feats_per_detector):
sets = []
for _ in range(num_base_detectors): #2
m1, m2 = get_highest_corr()
curr_set = [m1, m2] #3
for _ in range(2, num_feats_per_detector):
m = np.unravel_index(np.argsort(corr_matrix.values, axis=None),
corr_matrix.shape) #4
m0 = m[0][::-1]
m1 = m[1][::-1]
for i in range(len(m0)):
d0 = m0[i]
d1 = m1[i]
if (d0 in curr_set) or (d1 in curr_set): #5
curr_set.append(d0)
curr_set = list(set(curr_set))
if len(curr_set) < num_feats_per_detector:
curr_set.append(d1)
curr_set = list(set(curr_set)) #6
if len(curr_set) >= num_feats_per_detector:
break
for i in curr_set: #7
i_idx = corr_matrix.index[i]
for j in curr_set:
j_idx = corr_matrix.columns[j]
corr_matrix.loc[i_idx, j_idx] = 0
if len(curr_set) >= num_feats_per_detector:
break
sets.append(curr_set)
return sets
import pandas as pd
import numpy as np
from sklearn.datasets import fetch_openml
data = fetch_openml('baseball', version=1)
df = pd.DataFrame(data.data, columns=data.feature_names)
corr_matrix = abs(df.corr(method='spearman'))
corr_matrix = corr_matrix.where(
np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool))
corr_matrix = corr_matrix.fillna(0)
feat_sets_arr = get_correlated_subspaces(corr_matrix, num_base_detectors=5,
num_feats_per_detector=4)
for feat_set in feat_sets_arr:
print([df.columns[x] for x in feat_set])
The listing implements subspace selection from a feature-correlation matrix, starting with a highly correlated pair and adding associated features. section 10.1.2
