liveBook · Manning

Weighted average Gini impurity

As explained in Grokking Machine Learning, Second Edition

The weighted Gini index is the average Gini impurity of subsets produced by a split, weighted by their sample sizes.

When a decision tree splits data, each resulting subset may have a different mixture of labels. Weighted average Gini impurity combines those subset impurities while giving larger subsets more influence. section 9.9

The goal is to find a split with a small weighted value. A low score means the resulting leaves are relatively pure, so their predictions are more focused. section 9.9 section 9.6

Comparing age-based splits

For the split “Is the user younger than 14.5?”, the two groups contain ages 12 and 14 with labels {A, A}, and ages 15, 25, 32, and 35 with labels {A, C, B, C}. The first group is pure, while the second is mixed; their size-weighted Gini score is 0.417. The split at age 20 performs better, with a score of 0.222 and accuracy of 5/6. section 9.9

Table 9.4 The seven possible questions we can pick, each with the corresponding splitting. In the first set, we put the users who are younger than the cutoff, and in the second set, those who are older than the cutoff.
Question
First set (yes)
Second set (no)
Labels
Weighted accuracy
Weighted Gini impurity index
Weighted entropy
Is the user younger than 7?
empty
12, 14, 15, 25, 32, 35
{},
{A,A,A,C,B,C}
3/6
0.611
1.459
Is the user younger than 13?
12
14, 15, 25, 32, 35
{A},
{A,A,C,B,C}
3/6
0.533
1.268
Is the user younger than 14.5?
12, 14
15, 25, 32, 35
{A,A}
{A,C,B,C}
4/6
0.417
1.0
Is the user younger than 20?
12, 14, 15
25, 32, 35
{A,A,A},
{C,B,C}
5/6
0.222
0.459
Is the user younger than 28.5?
12, 14, 15, 25
32, 35
{A,A,A,C},
{B,C}
4/6
0.416
0.874
Is the user younger than 33.5?
12, 14, 15, 25, 32
35
{A,A,A,C,B},
{C}
4/6
0.467
1.145
Is the user younger than 100?
12, 14, 15, 25, 32, 35
empty
{A,A,A,C,B,C},
{}
3/6
0.611
1/459

Compare all seven possible age questions, their resulting label groups, weighted accuracies, weighted Gini impurity indexes, and weighted entropies. section 9.9

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Works through a numerical age-split example and compares seven candidate splits, but does not provide implementation code.
Examples
Uses app recommendations, user ages, labels A/B/C, and seven age-based questions as concrete split examples.
What it assumes
Assumes familiarity with decision-tree splits, labels, leaves, and basic proportions.
What you can do after
Evaluate candidate decision-tree splits and identify the one with the lowest weighted impurity.
How it connects
Connects decision trees with Gini impurity, Gini gain, entropy, information gain, accuracy, and leaf predictions.

Also explained in

Compare how 3 books explain weighted average Gini impurity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.