A univariate scoring method assigns an outlier score to each observation one feature at a time, then can combine feature scores to rank unusual rows. section 2.8
Use the method with these practical cautions:
- Sum scores only when their meanings are reasonably comparable; numeric and categorical features may require different tests. section 2.8
- Scale scores to a common range such as 0.0–1.0 when feature scales differ, while recognizing that equal weighting may distort importance. section 2.8
- Interpret row rankings cautiously: one extreme value may compete with several moderately unusual values, making the final judgment subjective. section 2.8
MAD scores on segment data
A Python implementation calculates median absolute deviation scores for each nonconstant column and aggregates them across rows. Region-centroid-row has a maximum score near 3.0, whereas vegde-sd reaches about 1750, illustrating why score scaling may matter. section 2.8
Listing 2.8 Creating outlier scores using univariate tests
import pandas as pd
import numpy as np
import statistics
from sklearn.preprocessing import MinMaxScaler
from sklearn.datasets import fetch_openml
def calc_MAD(data):
median = statistics.median(data)
deviations = [abs(x - median) for x in data]
median_deviation = statistics.median(deviations)
if median_deviation == 0: #1
mean_deviation = statistics.mean(deviations)
return [abs(x - median) / mean_deviation for x in data]
return [abs(x - median) / median_deviation for x in data]
data = fetch_openml('segment', version=1, parser='auto') #2
df = pd.DataFrame(data.data, columns=data.feature_names)
total_mad_scores = [0]*len(df)
for col_name in df.columns: #3
if df[col_name].nunique() == 1:
continue
mad_scores = calc_MAD(df[col_name])
mad_scores = np.array(mad_scores).reshape(-1, 1)
transformer = MinMaxScaler().fit(mad_scores) #4
col_mad_scores = transformer.transform(mad_scores).reshape(1, -1)[0]
total_mad_scores += col_mad_scores
print(total_mad_scores)
The listing provides a concise Python implementation using pandas, NumPy, statistics, scikit-learn, and the OpenML segment dataset. section 2.8
