liveBook · Manning

Random forest

As explained in Timeless Algorithms, The Foundational Ideas

A random forest is an ensemble model that combines predictions from multiple decision trees trained with randomized sampling or feature selection.

A single decision tree may fit its training data closely yet change substantially when that data changes. A random forest reduces this instability by combining many deliberately varied trees. Leo Breiman’s 2001 method extends bagging: it varies both the observations used to train each tree and the features considered at each split. section 9.0 section 9.3

A forest is trained through a repeated procedure:

  • For each tree, draw N observations with replacement from the N training observations. Some appear more than once; others are omitted. section 9.3.3
  • At each node, randomly select m of the p available features and choose the best split among them. Draw a new subset at each subsequent node; trees are usually grown deeply without pruning. section 9.3.3
  • Combine the trees’ predictions: use a majority vote for classification or an average for regression. section 9.3.3

For regression, the forest prediction is f̂ᴮ(x) = (1/B) ∑ᵦ₌₁ᴮ Tᵦ(x), where B is the number of trees and Tᵦ(x) is tree b’s prediction for observation x. Averaging reduces the influence of unusually high or low individual predictions; adding trees makes the average increasingly stable, though improvement eventually reaches a plateau. section 9.4.3

Diversity is essential to that stability. In ordinary bagging, a dominant predictor may shape the early splits of many trees, leaving their errors correlated. Randomly restricting candidate features can force different early splits and produce different tree structures. Individual trees may become weaker, but the forest can benefit if their errors become less correlated without sacrificing too much predictive strength. section 9.3 section 9.4

Figure 9.2 Feature subsampling in a random forest. The simplified example represents two trees trained to evaluate the same type of loan application. To isolate feature-level randomness, both trees are shown using the same training data. At the illustrated split, the first tree is permitted to consider only Age and Income, while the second can consider only Balance and Tenure. After evaluating the candidates available to it, the first tree selects Age ≤ 45 and the second selects Balance > X, where X represents the threshold that best satisfies the splitting criterion. These different initial decisions send applicants into different branches and influence every split that follows. In a complete random forest, each tree also receives a different bootstrap sample, and a new random subset of features is selected at every node. The trees and candidate sets shown here are deliberately simplified to isolate how feature selection creates structural diversity.

Two simplified loan-evaluation trees show how different candidate features can produce different initial splits, even with the same training data. section 9.3.2

A tree can use information gain to choose splits: predictors that repeatedly reduce entropy across trees can also receive high variable-importance scores. Observations omitted from bootstrap samples support out-of-bag evaluation, an internal estimate of predictive performance. Neither diagnostic removes the need to consider the quality of the data and the conditions in which predictions will be used. section 5.3.2 section 9.8 section 1.2.6

Random forests support classification and regression, particularly on structured data, and can capture nonlinear relationships and interactions. More trees do not fix shared errors, poor features, data leakage, or uncertainty about unseen observations. In lending, a stable risk prediction alone does not settle whether an approval decision is fair or appropriate. section 9.8 section 9.9 section 1.2.6

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Develops the training procedure and regression aggregation, then explains strength, correlation, convergence, and diagnostic measures.
Examples
Illustrates feature subsampling with two simplified loan-evaluation trees and considers repayment prediction in lending decisions.
What you can do after
Evaluate and understand how randomization, aggregation, tree correlation, and data quality affect forest predictions and their use.
How it connects
Connects random forests to decision trees, bagging, entropy, information gain, and support vector machines.

Also explained in

Compare how 17 books explain random forest →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.