liveBook · Manning

Information gain

As explained in Timeless Algorithms, The Foundational Ideas

Information gain is the reduction in uncertainty produced by a split or observation, commonly measured using entropy.

Information gain measures how much uncertainty a split removes. In a decision tree, it is the entropy before a candidate split minus the entropy remaining afterward; the tree favors splits with greater gain. section 5.3.1

1.00-0.80=0.20 bits

A split with 1.00 bit of initial entropy and 0.80 bits of weighted remaining entropy removes 0.20 bits of uncertainty. section 5.3.1

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines gain as the difference between pre-split and remaining entropy, then illustrates its calculation with a numerical example.
Examples
Calculates a 0.20-bit gain from 1.00 bit of initial entropy and 0.80 bits of weighted post-split entropy.
What you can do after
Evaluate candidate decision-tree splits by comparing their initial entropy with the uncertainty remaining after each split.
How it connects
Connects entropy to decision trees and relates information-theoretic partitioning to random forests, feature selection, and clustering.

Also explained in

Compare how 7 books explain information gain →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.