liveBook · Manning

Entropy

As explained in Grokking Machine Learning, Second Edition

A measure of uncertainty or impurity in a probability distribution, commonly used to evaluate classification splits.

Entropy measures how mixed the labels are in a set. A group with one label only has entropy 0, so it is perfectly pure. For two labels, entropy is 1 when the labels are evenly split, the point of greatest uncertainty. Lower entropy means a clearer label prediction. section 9.18

In a classification decision tree, entropy helps choose the next question. Compare the weighted average entropy of the groups created by each possible split, then prefer the split with the lower value. The drop in entropy from before to after a split is called information gain. section 9.18 section 9.3

Choosing an app recommendation question

Six users can be split by platform or age. The platform split has average entropy 0.918; the age split has average entropy 0.459 because all young users chose the same app. Age is the better first question because it leaves less label uncertainty. section 9.3

When groups have different sizes, weight each child’s entropy by its share of the original group. This prevents a tiny, pure group from outweighing a much larger mixed group. The same comparison works for thresholds on continuous features such as age. section 9.3 section 9.9

Using entropy in Scikit-Learn

A decision-tree display reports each node’s entropy, sample count, and label counts. Scikit-Learn uses the Gini index by default, but entropy can be selected with criterion='entropy' when creating a DecisionTreeClassifier. section 9.11

Figure 9.13 The two ways to split the dataset, by platform and age, and their entropy calculations. Notice that splitting the dataset by age gives us two smaller datasets with a lower average entropy. Therefore, we again choose to split the dataset by age.

Compare the two app-recommendation splits visually: the age split produces the lower average entropy. section 9.3

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Defines purity through endpoint values and works through weighted split comparisons, without deriving the entropy formula.
Examples
Uses app recommendations by age or platform and age-threshold questions with weighted entropy values.
What it assumes
Introduces entropy from label counts and simple split examples without assuming prior machine-learning knowledge.
What you can do after
Apply it via Scikit-Learn by selecting entropy as a DecisionTreeClassifier criterion and comparing tree splits.
How it connects
Connects to decision trees, information gain, Gini impurity index, accuracy, classification, and continuous features.

Also explained in

Compare how 19 books explain entropy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.