Mutual information
Mutual information is a measure of statistical dependence between random variables, quantifying the information about one variable provided by observing another.
Grokking Bayes
Mutual information is a measure of the dependence between random variables equal to the expected information gained about one variable from observing the other.
- How it is used
- Guides Bayesian experimental design by ranking observations according to their expected reduction of uncertainty.
- How deep it goes
- Defines dependence, independence, symmetry, and entropy connections, then stops short of deriving the probability formula.
- Examples
- Uses binary search, decision-tree splits, sensor placement, search and rescue, scientific experimentation, and active learning.
- What you can do after
- Evaluate experiments and observations by comparing their expected information about unknown quantities.
Timeless Algorithms, The Foundational Ideas
Mutual information is a measure of the reduction in uncertainty about one variable provided by observing another variable.
- How it is used
- Uses mutual information to rank predictors by how much they reduce uncertainty about a target during feature selection.
- How deep it goes
- Defines mutual information through entropy and gives a numerical calculation of uncertainty reduction.
- Examples
- Uses blood pressure and shoe size as contrasting disease predictors, alongside a numerical entropy calculation.
- What you can do after
- Evaluate predictors by comparing how much uncertainty each reduces about a target.
Read how Timeless Algorithms, The Foundational Ideas explains mutual information →
Also covered in
- Context EngineeringIn early access (MEAP)
- Machine Learning Algorithms in Depth
What to know first
- entropyEntropy is an information-theoretic measure of uncertainty in a probability distribution, defined as the expected value of the negative logarithm of outcome probability.
- posterior probabilityA posterior probability or distribution represents the probability of an event, hypothesis, or unknown quantity conditional on observed evidence and prior information.
Related concepts
- feature selectionused by · Feature selection is the process of choosing a subset of input features for use in a modeling, analysis, or prediction task.
- entropycontrasts with · Entropy is an information-theoretic measure of uncertainty in a probability distribution, defined as the expected value of the negative logarithm of outcome probability.
- information gainequivalent to · Information gain is the decrease in uncertainty resulting from information or an observation, often quantified as a reduction in entropy.
- maximum weight spanning treeconfigures · A maximum weight spanning tree is a spanning tree whose total edge weight is maximal among all spanning trees of a weighted graph.
- information retrievalmeasures · Information retrieval is the process of finding and selecting information relevant to a query or information need from a collection.
Questions about mutual information
What is mutual information?
Mutual information is a measure of statistical dependence between random variables, quantifying the information about one variable provided by observing another.
Which Manning books explain mutual information?
Grokking Bayes and Timeless Algorithms, The Foundational Ideas.
What should I know before learning mutual information?
Manning books that teach mutual information usually introduce entropy and posterior probability first.
What is the difference between mutual information and entropy?
Mutual information is a measure of statistical dependence between random variables, quantifying the information about one variable provided by observing another. Entropy is an information-theoretic measure of uncertainty in a probability distribution, defined as the expected value of the negative logarithm of outcome probability.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

