liveBook · Manning

Mutual information

As explained in Timeless Algorithms, The Foundational Ideas

Mutual information is a measure of the reduction in uncertainty about one variable provided by observing another variable.

For target Y and predictor X, mutual information is I(Y;X) = H(Y) − H(Y|X): the uncertainty about Y before observing X minus the uncertainty that remains afterward. In feature selection, predictors with higher mutual information reduce more uncertainty about the target. section 5.3.3

Disease prediction

Blood pressure may reduce uncertainty about a diagnosis, while shoe size is unlikely to change it. Mutual information quantifies this difference between predictors. section 5.3.3

I(Y;X)= H(Y)-H(Y|X)= 0.97- 0.85= 0.12 bits

The calculation subtracts conditional entropy from the target's original entropy, yielding 0.12 bits of mutual information. section 5.3.3

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines mutual information through entropy and gives a numerical calculation of uncertainty reduction.
Examples
Uses blood pressure and shoe size as contrasting disease predictors, alongside a numerical entropy calculation.
What you can do after
Evaluate predictors by comparing how much uncertainty each reduces about a target.
How it connects
Connects mutual information to entropy, feature selection, decision trees, and random forests.

Also explained in

Compare how 2 books explain mutual information →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.