How deeply does Grokking Machine Learning cover information gain?

Defines gain as entropy change and uses weighted split entropies, while deferring deeper information-theory treatment.

Information gain

As explained in Grokking Machine Learning

Information gain is the reduction or change in entropy resulting from a data split.

Information gain is the change in entropy produced by a data split. In a decision tree, compare the entropy before the split with the average entropy afterward; lower post-split entropy indicates a better split. section 9.8 section 9.2

Choosing a split for app recommendations

In a dataset of app downloads, splitting users by age gives lower average entropy than splitting by platform. If the resulting leaves differ in size, weight each leaf’s entropy by its share of the samples. section 9.2 section 9.2

Table 9.2 A simplified version of the dataset in table 9.1, where the age column has been simplified to two categories, “young” and “adult” (view table figure)

Platform	Age	App
iPhone	Young	Atom Count
iPhone	Adult	Check Mate Mate
Android	Adult	Beehive Finder
iPhone	Adult	Check Mate Mate
Android	Young	Atom Count
Android	Young	Atom Count

The app-recommendation dataset includes platform, age, and downloaded app—the fields used to illustrate candidate splits. section 9.2

How Grokking Machine Learning covers it

liveBook manning.com

Read Grokking Machine Learning on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Machine Learning for Tabular Data: XGBoost, Deep Learning, and AISplit gain is the improvement in a tree objective obtained by dividing a node with a candidate feature-based split.
liveBook manning.com

Grokking AI Algorithms, Second EditionInformation gain is the reduction in dataset uncertainty produced by applying a particular split or question.
liveBook manning.com

Statistics Every Programmer NeedsInformation gain is the reduction in uncertainty about a target produced by a partition or feature.

Compare how 4 books explain information gain →

Related concepts

entropymeasures · Entropy is a quantitative measure of uncertainty or dispersion in a probability distribution and, more generally, of disorder in the possible states of a system.
decision treeused by · A decision tree is a branching structure that represents sequential tests or choices, with paths leading to outcomes, predictions, or actions.
uncertaintymeasures · Uncertainty is incomplete knowledge or confidence about a value, state, interpretation, behavior, or outcome.
Kullback–Leibler divergenceequivalent to · Kullback–Leibler divergence is a nonnegative, directed measure of the information lost when one probability distribution approximates another, computed as an expected log ratio under the first distribution.

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning. The section links point to the parts of the book it draws on.