Information gain measures how much a partition reduces uncertainty about a target. Decision-tree algorithms use it to compare candidate feature splits for classification, while regression trees use error-based criteria such as mean squared error or mean absolute error. section 6.1
For a classification split, entropy measures impurity or disorder before and after partitioning; information gain is the resulting reduction in entropy. Higher reduction indicates a more informative split. Gini impurity evaluates split quality differently and commonly computes faster, while entropy can be more sensitive to changes in class probabilities. section 6.3.2
Use information gain within a practical tree-building workflow:
- Compare candidate feature splits by their reduction in classification uncertainty. section 6.1
- Select entropy as the decision-tree criterion when using an information-gain-based impurity measure. section 6.3.2
- Control tree complexity with parameters such as max_depth to reduce overfitting while building the tree. section 6.3.2
