Entropy measures how mixed the labels are in a set. A group with one label only has entropy 0, so it is perfectly pure. For two labels, entropy is 1 when the labels are evenly split, the point of greatest uncertainty. Lower entropy means a clearer label prediction. section 9.18
In a classification decision tree, entropy helps choose the next question. Compare the weighted average entropy of the groups created by each possible split, then prefer the split with the lower value. The drop in entropy from before to after a split is called information gain. section 9.18 section 9.3
Choosing an app recommendation question
Six users can be split by platform or age. The platform split has average entropy 0.918; the age split has average entropy 0.459 because all young users chose the same app. Age is the better first question because it leaves less label uncertainty. section 9.3
When groups have different sizes, weight each child’s entropy by its share of the original group. This prevents a tiny, pure group from outweighing a much larger mixed group. The same comparison works for thresholds on continuous features such as age. section 9.3 section 9.9
Using entropy in Scikit-Learn
A decision-tree display reports each node’s entropy, sample count, and label counts. Scikit-Learn uses the Gini index by default, but entropy can be selected with criterion='entropy' when creating a DecisionTreeClassifier. section 9.11
Figure 9.13 The two ways to split the dataset, by platform and age, and their entropy calculations. Notice that splitting the dataset by age gives us two smaller datasets with a lower average entropy. Therefore, we again choose to split the dataset by age.
Compare the two app-recommendation splits visually: the age split produces the lower average entropy. section 9.3


















