Entropy measures disorder or uncertainty in a probability distribution or dataset. In a decision tree, uncertainty describes how mixed the labels are before and after a question splits the examples. section 8.17
Decision-tree training uses uncertainty to choose useful questions:
- Count how many examples belong to each class. section 8.17
- Try feature-value questions such as “Is Carat >= 0.5?” and split examples into true and false groups. section 8.17
- Prefer the question that produces the greatest information gain, meaning the largest reduction in uncertainty. section 8.17
Diamond classification
A tree can test Carat and Price values to separate Okay and Perfect diamonds. The resulting branches continue splitting until a leaf contains examples from one class or no further gain is available. section 8.17
Figure 8.20 Example of uncertainty
Use the uncertainty illustration to visualize why mixed class groups are less certain than single-class groups. section 8.17
