Median absolute deviation (MAD) measures statistical spread by taking the median of the absolute distances from a dataset’s median. It is more robust to outliers than the IQR. section 2.1.3
Calculate and interpret MAD scores in three steps:
- Find the data median, calculate each point’s absolute deviation from it, then take the median of those deviations as the MAD. section 2.1.3
- Score each point as absolute deviation divided by MAD; larger scores indicate greater distance from the typical deviation. section 2.1.3
- Flag scores above a chosen threshold, often 3.0, while treating the threshold as adjustable rather than universal. section 2.1.3
Abalone whole-weight data
Applying MAD scoring to the abalone dataset makes an inserted value of 4.0 stand out, receiving a score slightly above 8.5. The resulting score distribution shows progressively larger scores as values move farther from the median. section 2.1.3
MAD is primarily a univariate detector for numeric columns. It can also test unusually small category counts, but few distinct values can make that approach unreliable. Combining MAD with cumulative-frequency checks can improve categorical flagging. section 2.2 section 2.9
When aggregating scores across columns, compare scales cautiously: numeric and categorical tests may produce different meanings, and even comparable MAD scores can make row-level judgments subjective. section 2.8
Listing 2.1 Calculating the MAD
import pandas as pd
import numpy as np
import statistics
from sklearn.datasets import fetch_openml
import matplotlib.pyplot as plt
def calc_MAD(data): #1
median = statistics.median(data)
deviations = [abs(x - median) for x in data]
median_deviation = statistics.median(deviations)
mad_scores = [abs(x - median) / median_deviation for x in data]
return mad_scores
data = fetch_openml("abalone", version=1, parser='auto') #2
data = pd.DataFrame(data.data, columns=data.feature_names)
fig, ax = plt.subplots(nrows=1, ncols=3, figsize=(10, 3))
pd.Series(data['Whole_weight']).hist(bins=50, ax=ax[0]) #3
ax[0].set_title("Whole_weight")
mad_scores = calc_MAD(data['Whole_weight']) #4
pd.Series(mad_scores).hist(bins=50, ax=ax[1])
ax[1].set_title("Distribution of MAD Scores")
mad_scores = calc_MAD(np.concatenate([data['Whole_weight'], [4.0]])) #5
pd.Series(mad_scores).hist(bins=50, ax=ax[2]) #6
ax[2].set_title("MAD Scores given an outlier")
plt.tight_layout()
plt.show()
Inspect a concise Python implementation that calculates medians, deviations, MAD scores, and score distributions on abalone data. section 2.1.3
