Information gain
Information gain is the reduction in uncertainty about a target resulting from new information, commonly measured as a decrease in entropy.
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
Split gain is the improvement in a tree objective obtained by dividing a node with a candidate feature-based split.
- How it is used
- Guides classification-tree split selection by favoring feature values that reduce label uncertainty.
- How deep it goes
- Defines information gain intuitively and connects it to split search, LightGBM growth, and histogram-based XGBoost split finding without a derivation.
- Examples
- Uses animal-tree figures, an Airbnb NYC classifier, leaf-wise versus level-wise growth, and a latitude histogram-split timing comparison.
- What you can do after
- Evaluate/understand why a tree selects particular classification splits and why maximum-gain-first growth changes constrained tree construction.
Grokking Machine Learning
Information gain is the reduction or change in entropy resulting from a data split.
- How it is used
- Uses entropy changes to compare candidate splits when building a decision tree.
- How deep it goes
- Defines gain as entropy change and uses weighted split entropies, while deferring deeper information-theory treatment.
- Examples
- Compares age and platform splits in an app-recommendation dataset, with age producing lower average entropy.
- What you can do after
- Evaluate candidate decision-tree splits by comparing average entropy and weighting leaves by sample share when sizes differ.
Read how Grokking Machine Learning explains information gain →
Grokking AI Algorithms, Second Edition
Information gain is the reduction in dataset uncertainty produced by applying a particular split or question.
- How it is used
- Ranks candidate questions while training decision trees.
- How deep it goes
- Explains the weighted split calculation and implements exhaustive feature-value search, without developing formal proofs.
- Examples
- Uses a diamond-price threshold and a split yielding Perfect diamonds on one side.
- What you can do after
- Build a Python split scorer that selects the highest-gain question for a decision-tree node.
Read how Grokking AI Algorithms, Second Edition explains information gain →
Statistics Every Programmer Needs
Information gain is the reduction in uncertainty about a target produced by a partition or feature.
- How it is used
- Guides classification-tree feature splits by measuring reductions in target uncertainty.
- How deep it goes
- Defines entropy-based reduction and contrasts it with Gini impurity, but does not derive the entropy formula.
- Examples
- Uses entropy as a DecisionTreeClassifier criterion and contrasts it with the default Gini criterion.
- What you can do after
- Apply it via decision-tree criteria to evaluate classification splits and configure model complexity.
Read how Statistics Every Programmer Needs explains information gain →
Related concepts
- entropymeasures · Entropy is a quantitative measure of uncertainty or dispersion in a probability distribution and, more generally, of disorder in the possible states of a system.
- decision treeused by · A decision tree is a branching structure that represents sequential tests or choices, with paths leading to outcomes, predictions, or actions.
- uncertaintymeasures · Uncertainty is incomplete knowledge or confidence about a value, state, interpretation, behavior, or outcome.
- Kullback–Leibler divergenceequivalent to · Kullback–Leibler divergence is a nonnegative, directed measure of the information lost when one probability distribution approximates another, computed as an expected log ratio under the first distribution.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.