Mutual information measures how much knowing one random variable tells us about another. It is nonnegative, equals zero for independent variables, and is largest when one completely determines the other. section 13.10
Its symmetry gives two equivalent views: information about an unknown quantity gained from an observation, or uncertainty about the observation reduced by knowing the unknown quantity. section 13.10
Mutual information supports several practical decisions:
- Select experiments whose observations are expected to reduce uncertainty about unknown parameters. section 13.10
- Choose decision-tree splits that provide the most information about the predicted label. section 13.10
- Replace repeated posterior and entropy calculations with forward-model computations when evaluating expected information gain. section 13.10
Binary search
Binary search illustrates the same principle: splitting the search space equally in the middle maximizes the information gained at each step. section 13.11
Express expected information gain as conditional mutual information between the unknown quantity θ and observation y given experiment x. section 13.10

