Entropy measures uncertainty in a probability distribution. In outlier detection, it is mainly applied to categorical data: each cell value receives a univariate entropy score, and the cell scores combine into a row score without considering associations between features. section 7.9
For a categorical feature, evenly distributed values produce higher entropy because any value is difficult to predict; concentrated values produce lower entropy. Numeric data can be handled, but usually requires binning first. Entropy also relates to decision trees, where splits seek purer child nodes, although Gini impurity is often preferred because logarithms are slower than squaring. section 7.9
Three color features
A feature containing 100 red, 100 blue, and 100 green values has high entropy. Six equally common colors have higher entropy still, while a feature containing 290 red and 10 blue values has low entropy. section 7.9
Use entropy-based detection with these practical constraints:
- Treat it as a categorical detector in most applications, binning numeric features before scoring. section 7.9
- Interpret results as primarily univariate: the method does not model associations between features. section 7.9
- Consider it alongside other detectors; it can support categorical outlier detection but is not a universal solution. section 7.14
Listing 7.11 Example of outlier detection using Entropy
class EntropyOutlierDetector:
def __init__(self, num_outliers, num_bins=7):
self.num_outliers = int(num_outliers)
self.num_bins = num_bins
def calculate_entropy(self, values): #1
vc = values.value_counts(normalize=True)
entropy = 0.0
for v in vc.values:
if v > 0:
entropy += ((v) * np.log2(v))
entropy = (-1) * entropy
return entropy
def fit_predict(self, df):
df = df.copy()
df['A binned'] = pd.cut(
df[df.columns[0]], bins=self.num_bins) #2
df['B binned'] = pd.cut(
df[df.columns[1]], bins=self.num_bins)
temp_df = df.copy()
scores = [0]*len(df)
for iteration_num in range(self.num_outliers): #3
lowest_entropy = np.inf
lowest_entropy_row = -1
for i in temp_df.index: #4
a_entropy = self.calculate_entropy(temp_df['A binned'].drop(i))
b_entropy = self.calculate_entropy(temp_df['B binned'].drop(i))
total_entropy = a_entropy + b_entropy #5
if total_entropy < lowest_entropy:
lowest_entropy = total_entropy
lowest_entropy_row = i
scores[lowest_entropy_row] = (self.num_outliers - iteration_num)
temp_df = temp_df.drop(index=lowest_entropy_row)
return scores
df = create_four_clusters_test_data()
df = pd.DataFrame(RobustScaler().fit_transform(df), columns=df.columns)
df['Entropy Scores'] = EntropyOutlierDetector(num_outliers=10, num_bins=10).fit_predict(df)
Shows a compact entropy detector implementation with configurable outlier count, numeric binning, entropy calculation, and iterative row selection. section 7.9
Figure 7.10 The output of the entropy detector on the test datasets. Here we can see quite different results than previous detectors as the algorithm works very differently, focusing on univariate outliers.
Illustrates how entropy detection can produce results unlike other detectors because it focuses on univariate outliers. section 7.9
