Suppose you can ask one question before recommending an app. Would you ask a user’s age or phone platform? A useful question separates users into groups whose app choices are less mixed. Entropy puts a number on that mixture: a group with one label has entropy 0; a group split evenly between two labels has entropy 1. section 9.2 section 9.8
To compare questions, split the labeled examples by each possible answer, calculate each group’s entropy, and average the results. Weight each group by its share of the examples: a group of six should count more than a group of two. Prefer the split with lower weighted entropy. The drop from the original group’s entropy to the split’s weighted entropy is called information gain. section 9.2 section 9.8
Age or platform?
Six users have downloaded one of three apps. Splitting by platform gives two mixed groups, each with entropy 0.918, so the average is 0.918. Splitting by age gives one pure group with entropy 0 and one mixed group with entropy 0.918. Because the groups are equally sized, the average is 0.459. Age is the better first question. The remaining mixed group can then be split by platform into pure groups. section 9.2
Figure 9.13 The two ways to split the dataset, by platform and age, and their entropy calculations. Notice that splitting the dataset by age gives us two smaller datasets with a lower average entropy. Therefore, we again choose to split the dataset by age.
Compare the two candidate splits and their average entropies side by side. section 9.2
Entropy can also compare yes-or-no questions made from a categorical feature or thresholds on a numerical feature. It is one choice alongside accuracy and Gini impurity for selecting classification-tree splits. In Scikit-Learn, `DecisionTreeClassifier(criterion='entropy')` selects it as the splitting criterion. section 9.3 section 9.4
Try choosing the first question in a fraud-detection tree using transaction value and approved-vendor status. Compare the candidate splits using entropy, then repeat using Gini impurity: do they select the same question? section 9.9
