liveBook · Manning

Mutual information

Mutual information is a measure of statistical dependence between random variables, quantifying the information about one variable provided by observing another.

How 2 Manning books explain it

Grokking Bayes

Mutual information is a measure of the dependence between random variables equal to the expected information gained about one variable from observing the other.

How it is used
Guides Bayesian experimental design by ranking observations according to their expected reduction of uncertainty.
How deep it goes
Defines dependence, independence, symmetry, and entropy connections, then stops short of deriving the probability formula.
Examples
Uses binary search, decision-tree splits, sensor placement, search and rescue, scientific experimentation, and active learning.
What you can do after
Evaluate experiments and observations by comparing their expected information about unknown quantities.

Read how Grokking Bayes explains mutual information →

Timeless Algorithms, The Foundational Ideas

Mutual information is a measure of the reduction in uncertainty about one variable provided by observing another variable.

How it is used
Uses mutual information to rank predictors by how much they reduce uncertainty about a target during feature selection.
How deep it goes
Defines mutual information through entropy and gives a numerical calculation of uncertainty reduction.
Examples
Uses blood pressure and shoe size as contrasting disease predictors, alongside a numerical entropy calculation.
What you can do after
Evaluate predictors by comparing how much uncertainty each reduces about a target.

Read how Timeless Algorithms, The Foundational Ideas explains mutual information →

Also covered in

What to know first

Related concepts

Questions about mutual information

What is mutual information?

Mutual information is a measure of statistical dependence between random variables, quantifying the information about one variable provided by observing another.

Which Manning books explain mutual information?

Grokking Bayes and Timeless Algorithms, The Foundational Ideas.

What should I know before learning mutual information?

Manning books that teach mutual information usually introduce entropy and posterior probability first.

What is the difference between mutual information and entropy?

Mutual information is a measure of statistical dependence between random variables, quantifying the information about one variable provided by observing another. Entropy is an information-theoretic measure of uncertainty in a probability distribution, defined as the expected value of the negative logarithm of outcome probability.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.