liveBook · Manning

Feature selection

As explained in Timeless Algorithms, The Foundational Ideas

Feature selection identifies a subset of predictors that contributes useful information while excluding predictors that add little value.

Feature selection chooses predictors that provide useful information about a target while excluding those that add little, duplicate stronger predictors, or increase complexity in ways that can weaken generalization. Mutual information quantifies how much a predictor reduces uncertainty about the target. section 5.3.3

I(Y;X)= H(Y)-H(Y|X)= 0.97- 0.85= 0.12 bits

This calculation shows mutual information as the target’s original uncertainty minus its remaining uncertainty after observing a predictor. section 5.3.3

Disease prediction

Blood pressure may reduce uncertainty about a diagnosis, while shoe size is unlikely to change the prediction. Mutual information provides a quantitative way to compare their value. section 5.3.3

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines mutual information and illustrates predictor value with a numerical entropy calculation and medical example.
Examples
Illustrates predictor value with blood pressure and shoe size in disease prediction, plus a mutual-information calculation.
What you can do after
Evaluate predictors by comparing how much uncertainty each reduces about a target.
How it connects
Connects feature selection to entropy, mutual information, decision trees, random forests, and model generalization.

Also explained in

Compare how 11 books explain feature selection →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.