How deeply does Grokking AI Algorithms, Second Edition cover information gain?

Explains the weighted split calculation and implements exhaustive feature-value search, without developing formal proofs.

Information gain

As explained in Grokking AI Algorithms, Second Edition

Information gain is the reduction in dataset uncertainty produced by applying a particular split or question.

Information gain measures how much a question reduces uncertainty in a dataset. Decision trees compare candidate feature questions and prefer the split producing the greatest reduction. section 8.17

Calculate it by comparing uncertainty before and after a split:

Split the examples using a question, such as whether Price is at least 3,914. section 8.17
Measure the Gini index for the left and right groups, weighting each group's uncertainty by its size. section 8.17
Select the question with the highest gain as the next decision node. section 8.17

A useful split

For the question “Is Price >= 3,914?”, calculate the gain for its two resulting groups. A split that leaves one side containing only Perfect diamonds has reduced uncertainty substantially, making the question a strong candidate. section 8.17

How Grokking AI Algorithms, Second Edition covers it

Cover of Grokking AI Algorithms, Second Edition

liveBook manning.com

Read Grokking AI Algorithms, Second Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Machine Learning for Tabular Data: XGBoost, Deep Learning, and AISplit gain is the improvement in a tree objective obtained by dividing a node with a candidate feature-based split.
liveBook manning.com

Grokking Machine LearningInformation gain is the reduction or change in entropy resulting from a data split.
liveBook manning.com

Statistics Every Programmer NeedsInformation gain is the reduction in uncertainty about a target produced by a partition or feature.

Compare how 4 books explain information gain →

Related concepts

entropymeasures · Entropy is a quantitative measure of uncertainty or dispersion in a probability distribution and, more generally, of disorder in the possible states of a system.
decision treeused by · A decision tree is a branching structure that represents sequential tests or choices, with paths leading to outcomes, predictions, or actions.
uncertaintymeasures · Uncertainty is incomplete knowledge or confidence about a value, state, interpretation, behavior, or outcome.
Kullback–Leibler divergenceequivalent to · Kullback–Leibler divergence is a nonnegative, directed measure of the information lost when one probability distribution approximates another, computed as an expected log ratio under the first distribution.

This explainer was generated by Manning's knowledge graph from the text of Grokking AI Algorithms, Second Edition. The section links point to the parts of the book it draws on.