liveBook · Manning

Random feature selection

As explained in Timeless Algorithms, The Foundational Ideas

Random feature selection restricts the features considered at a decision-tree split to a randomly chosen subset.

Random feature selection limits the predictors a decision tree can consider at each split. In a random forest, it helps prevent a strong predictor from shaping every tree in the same way, making aggregation more effective. section 9.2.3 section 9.3.2

At each node, draw m features without replacement from the full set of p features. Evaluate possible splits using only those candidates, choose the best according to the tree’s splitting criterion, and repeat at the resulting nodes. The value of m stays fixed during training, but a new subset is drawn at every node. A feature excluded at one split can reappear later. section 9.3.1 section 9.3.3

The restriction changes the candidates, not the underlying tree method. Classification trees seek reductions in class impurity; regression trees seek reductions in numerical variation. Feature selection also differs from bootstrap sampling: the latter draws observations with replacement for each tree, while the former draws features without replacement for each node. section 9.3.1

Figure 9.2 Feature subsampling in a random forest. The simplified example represents two trees trained to evaluate the same type of loan application. To isolate feature-level randomness, both trees are shown using the same training data. At the illustrated split, the first tree is permitted to consider only Age and Income, while the second can consider only Balance and Tenure. After evaluating the candidates available to it, the first tree selects Age ≤ 45 and the second selects Balance > X, where X represents the threshold that best satisfies the splitting criterion. These different initial decisions send applicants into different branches and influence every split that follows. In a complete random forest, each tree also receives a different bootstrap sample, and a new random subset of features is selected at every node. The trees and candidate sets shown here are deliberately simplified to isolate how feature selection creates structural diversity.

The loan-application example isolates feature randomness: different candidate sets lead two trees using the same data to choose different initial splits. section 9.3.2

Different early splits send observations into different groups, so later splits can diverge as well. This diversity can reduce correlation among tree errors, allowing voting in classification or averaging in regression to stabilize predictions. But restricting candidates can also weaken individual trees. A useful forest balances predictive strength against correlation; diversity alone is not enough. section 9.3.2 section 9.4 section 9.9

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Details candidate selection and recursive splitting, then explains the trade-off between tree strength and correlation.
Examples
Illustrates different candidate sets and initial splits with two simplified loan-application trees trained on the same data.
What you can do after
Evaluate and understand how feature randomness can reduce correlated errors while potentially weakening individual trees.
How it connects
Connects decision trees, bootstrap sampling, bagging, aggregation, and generalization.

Also explained in

Compare how 2 books explain random feature selection →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.