liveBook · Manning

Information gain

As explained in Grokking Machine Learning, Second Edition

The reduction in entropy obtained by comparing a parent node with the weighted entropy of its child leaves.

Information gain measures how much uncertainty a decision-tree split removes. Compare the parent node’s entropy with the weighted entropy of its child leaves: a larger reduction indicates a better split. section 9.18 section 9.6

Use the idea in three steps:

  • Measure the parent node’s entropy. A completely uniform label set has entropy 0; a balanced two-label set has entropy 1. section 9.18
  • Split the samples and calculate each leaf’s entropy, weighting each result by that leaf’s share of the samples. section 9.3
  • Prefer the split with the greatest entropy reduction, or equivalently the lowest weighted entropy when the parent stays fixed. section 9.6

Choosing age over platform

For six app users, splitting by platform gives an average entropy of 0.918. Splitting by age gives 0.459 because all young users share one label. Since age leaves less remaining uncertainty, age provides greater information gain.

Table 9.2 A simplified version of the dataset in table 9.1, where the age column has been simplified to two categories, “young” and “adult”
Platform
Age
App
iPhone
Young
Atom Count
iPhone
Adult
Check Mate Mate
Android
Adult
Beehive Finder
iPhone
Adult
Check Mate Mate
Android
Young
Atom Count
Android
Young
Atom Count

Inspect the small platform-and-age dataset behind the entropy comparison, including its app labels. section 9.3

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Explains entropy reduction, weighted child averages, and split comparison, while deferring detailed information-theory treatment.
Examples
Compares platform and age splits in an app-recommendation dataset using average entropies of 0.918 and 0.459.
What it assumes
Assumes basic familiarity with labeled datasets and decision-tree nodes, while explaining entropy and weighted averages from their roles.
What you can do after
Apply it via decision-tree training to choose splits that leave the child leaves with less weighted entropy.
How it connects
Connects information gain with decision trees, entropy, information theory, classification, and Gini impurity.

Also explained in

Compare how 7 books explain information gain →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.