Feature selection chooses predictors that provide useful information about a target while excluding those that add little, duplicate stronger predictors, or increase complexity in ways that can weaken generalization. Mutual information quantifies how much a predictor reduces uncertainty about the target. section 5.3.3
This calculation shows mutual information as the target’s original uncertainty minus its remaining uncertainty after observing a predictor. section 5.3.3
Disease prediction
Blood pressure may reduce uncertainty about a diagnosis, while shoe size is unlikely to change the prediction. Mutual information provides a quantitative way to compare their value. section 5.3.3










