liveBook · Manning

Mutual information

As explained in Grokking Bayes

Mutual information is a measure of the dependence between random variables equal to the expected information gained about one variable from observing the other.

Mutual information measures how much knowing one random variable tells us about another. It is nonnegative, equals zero for independent variables, and is largest when one completely determines the other. section 13.10

Its symmetry gives two equivalent views: information about an unknown quantity gained from an observation, or uncertainty about the observation reduced by knowing the unknown quantity. section 13.10

Mutual information supports several practical decisions:

  • Select experiments whose observations are expected to reduce uncertainty about unknown parameters. section 13.10
  • Choose decision-tree splits that provide the most information about the predicted label. section 13.10
  • Replace repeated posterior and entropy calculations with forward-model computations when evaluating expected information gain. section 13.10

Binary search

Binary search illustrates the same principle: splitting the search space equally in the middle maximizes the information gained at each step. section 13.11

EIG(𝑥) = 𝐼(𝜃; 𝑦 | 𝑥).

Express expected information gain as conditional mutual information between the unknown quantity θ and observation y given experiment x. section 13.10

How Grokking Bayes covers it

How deep it goes
Defines dependence, independence, symmetry, and entropy connections, then stops short of deriving the probability formula.
Examples
Uses binary search, decision-tree splits, sensor placement, search and rescue, scientific experimentation, and active learning.
What it assumes
Builds on probability, entropy, expected information gain, and Bayesian updating while treating dependence intuitively.
What you can do after
Evaluate experiments and observations by comparing their expected information about unknown quantities.
How it connects
Connects mutual information with entropy, expected information gain, KL divergence, Bayesian experimental design, and decision trees.

Also explained in

Compare how 2 books explain mutual information →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Bayes. The section links point to the parts of the book it draws on.