Information gain

As explained in Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI

Split gain is the improvement in a tree objective obtained by dividing a node with a candidate feature-based split.

Information gain measures how much a candidate split reduces uncertainty about class labels. In a classification tree, the algorithm searches feature values for splits that leave more homogeneous target groups; the split with the strongest improvement is preferred. section 5.1

It is one of several ways to judge classification splits. Entropy describes label disorder, information gain expresses the uncertainty reduction produced by a split, and Gini impurity estimates random-label misclassification risk. Regression trees instead seek splits that reduce target error or variation, such as mean squared error, mean absolute error, or variance. section 5.1

Information gain guides tree construction in several ways:

• Search candidate features and observed values, then select the feature-value combination producing the most homogeneous class subsets. section 5.1
• Prioritize the leaf with maximum loss reduction in LightGBM’s leaf-wise growth, rather than splitting every node at one depth. section 5.5.1
• Evaluate candidate thresholds efficiently with presorting or histograms when finding XGBoost splits. section 5.4.3

Choosing a tree-growing strategy

With a limit on depth or terminal leaves, leaf-wise growth repeatedly expands the leaf with the greatest information gain first. This can yield smaller trees, faster training, and higher accuracy than level-wise growth, though the approaches differ chiefly because the stopping rule interrupts their construction paths. section 5.5.1

Figure 5.10 How level-wise (on the left) and leaf-wise (on the right) tree growth differ

Compare level-wise and leaf-wise growth paths to see why selecting the maximum-gain leaf changes which splits appear before a tree reaches its size limit. section 5.5.1

Accelerating threshold search

For each node, XGBoost can enumerate features, inspect values, compute information gain for candidate splits, and retain the best result. Histogram-based candidates reduce the number of split points examined and can be evaluated in parallel; the illustrated latitude split produces the same threshold with a much shorter reported runtime when using 256 bins. section 5.4.3 section 5.4.3

How Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI covers it

How deep it goes

Defines information gain intuitively and connects it to split search, LightGBM growth, and histogram-based XGBoost split finding without a derivation.

Examples

Uses animal-tree figures, an Airbnb NYC classifier, leaf-wise versus level-wise growth, and a latitude histogram-split timing comparison.

What you can do after

Evaluate/understand why a tree selects particular classification splits and why maximum-gain-first growth changes constrained tree construction.

How it connects

Connects entropy, Gini impurity, decision trees, regression criteria, LightGBM, XGBoost, histogram splitting, and loss reduction.

liveBookmanning.com

Read Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningInformation gain is the reduction or change in entropy resulting from a data split.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionInformation gain is the reduction in dataset uncertainty produced by applying a particular split or question.
• 

  liveBookmanning.com
  Statistics Every Programmer NeedsInformation gain is the reduction in uncertainty about a target produced by a partition or feature.

Compare how 4 books explain information gain →

Related concepts

• entropymeasures · Entropy is a quantitative measure of uncertainty or dispersion in a probability distribution and, more generally, of disorder in the possible states of a system.
• decision treeused by · A decision tree is a branching structure that represents sequential tests or choices, with paths leading to outcomes, predictions, or actions.
• uncertaintymeasures · Uncertainty is incomplete knowledge or confidence about a value, state, interpretation, behavior, or outcome.
• Kullback–Leibler divergenceequivalent to · Kullback–Leibler divergence is a nonnegative, directed measure of the information lost when one probability distribution approximates another, computed as an expected log ratio under the first distribution.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI. The section links point to the parts of the book it draws on.