Chapter 9: From single trees to random forests: Leo Breiman and the logic of ensemble learning
This introduction explains why unstable decision trees can benefit from an ensemble and contrasts Breiman’s approach to generalization with Vapnik’s.
After this chapter you will be able to
- Single tree instability. Explain why a decision tree can fit its training data yet fail to generalize reliably.
- Diversity through aggregation. Explain Breiman’s central idea of combining deliberately different trees rather than relying on one tree.
- Tree splits and predictions. Trace how a decision tree recursively splits training observations and predicts for a new observation. Distinguish the split criteria and leaf predictions used for classification and regression.
- Single-tree instability and overfitting. Explain how early splits and increasing depth can make a single tree unreliable on new observations.
- Early split instability. Trace how a small sample change can alter a near-tied split, particularly near the root, and thereby change the data and split choices in downstream nodes.
- Depth and overfitting. Explain why splits based on very small groups can fit training noise and make training performance misleading about performance on new data.
- Variance and depth trade-off. Describe why deep trees have high variance and explain the trade-off when their depth is restricted.
- Limits of one pruned tree. Explain why pruning cannot eliminate instability and what flexibility a tree gives up when it is restricted.
- Stability through varied trees. Explain why combining predictions from varied trees offers an alternative to making one tree stable, and state the basic bagging approach introduced here.
- Bagging and tree stability. Explain why decision trees are well suited to bagging and how combining trees trained on varied observation samples stabilizes predictions without making each tree less expressive.
- Bootstrap samples within bagging. Describe how bagging begins with bootstrap sampling: repeatedly draw observations with replacement to make training samples of the original size, then train a model on each sample and aggregate predictions.
- Sampling effects on tree structure. Explain how training separate trees on bootstrap samples can change their splits and structures even though the tree-training method and available features stay the same.
- Classification votes and regression averages. Describe how bagging produces one prediction from trees trained on bootstrap samples: it selects the most-voted class for classification and averages numerical predictions for regression.
- Ensemble stability through aggregation. Explain why combining predictions reduces variance even though individual bootstrap-trained trees remain sensitive to their samples.
- Bagging and variance reduction. Explain how bootstrap samples change the trees that are trained and how combining their predictions reduces the influence of any one sample or tree.
- Correlated errors limit aggregation. Explain why bootstrap sampling alone may leave trees too similar and why adding more such trees offers diminishing benefit.
- Random forests add feature randomness. Describe the additional source of diversity random forests introduce beyond bootstrap sampling and why it addresses a limit of bagging.
- Two sources of forest randomness. Describe how a random forest combines bootstrap samples with a newly selected random subset of features at each node, chooses the best split from that subset, and aggregates predictions across trees.
- Diversity and aggregation tradeoff. Explain why withholding strong predictors at some splits can make individual trees slightly less accurate yet improve the aggregated forest by making tree structures and errors less correlated.
- Fresh feature subsets at splits. Explain how a random forest draws a new subset of m out of p features at every node and chooses the best split available within that subset. Describe how the rest of the decision-tree process remains the same.
- Two sources of tree variation. Distinguish bootstrap sampling of observations from random feature selection, and explain why adding feature randomness can make the trees follow different paths rather than repeatedly favoring dominant predictors.
- Feature limits and tree diversity. Trace how withholding a dominant predictor at some splits leads trees to choose other useful predictors and develop different structures, without permanently removing the dominant predictor.
- Diversity and error correlation. Explain why structurally different trees are less likely to make the same errors and how that makes voting or averaging more effective at reducing variance.
- Feature subset size trade-off. Explain why feature subsets that are too small can weaken individual trees, while subsets that are too large can leave trees overly similar.
- Randomized tree construction. Trace how a random forest trains many trees: draw a new bootstrap sample for each tree, restrict the candidate features separately at each split, and grow trees using ordinary recursive splitting.
- Forest prediction aggregation. Describe how the trained trees produce one forest prediction for classification and for regression.
- Tree strength and diversity. Explain why reliable generalization requires trees that predict better than chance without repeatedly making the same errors, and how random feature selection helps balance those requirements.
- Classification margin from votes. Explain how to calculate a classification margin for an observation with a known class and interpret its sign and size.
- Margin distribution and error. Explain how margins across labeled observations relate to classification error and what their distribution reveals beyond accuracy.
- Classification strength and correlation. Explain how forest strength is defined from classification margins and why a forest needs both useful trees and differences in their errors.
- Regression averaging and shared errors. Explain why averaging regression trees can reduce variance when their errors are not closely correlated, and why diversity cannot remove shared bias.
- Candidate feature trade-off. Explain how the number of candidate features at each split trades individual tree quality against similarity among trees, and how forests can be compared to choose a balance.
- Tree Depth Versus Tree Count. Distinguish the effect of deepening one decision tree from adding another tree to a random forest. Explain that an added tree is separately randomized and equally weighted rather than making existing trees more complex.
- Convergence of Forest Predictions. Explain how predictions stabilize as the number of randomized trees grows: regression averages approach an expected prediction, while classification vote proportions approach expected vote shares. Describe why error tends to plateau rather than systematically worsen merely because more trees are added.
- Limits of a Stable Forest. Explain why convergence does not guarantee accurate predictions or prevent a forest from performing poorly. Relate the quality of the stable result to tree strength and correlation, and state why judging it requires observations not used to train the relevant trees.
- Out-of-bag prediction construction. Trace how a training observation receives an out-of-bag prediction from only the trees whose bootstrap samples omitted it, using votes for classification or an average for regression.
- Internal validation and error measures. Explain why out-of-bag predictions provide internal validation and describe how they are compared with known outcomes for classification or regression.
- Forest-wide feature importance. Explain why interpreting a random forest requires ensemble-level measures rather than tracing one tree, and describe the two broad approaches introduced here, including the role of out-of-bag observations.
- Permutation importance calculation. Trace how permutation importance compares baseline out-of-bag error with error after shuffling one feature, without retraining the forest.
- Impurity reduction accounting. Describe how impurity-based importance credits features for weighted split improvements across a forest and what a normalized score represents.
- Importance measures and limitations. Distinguish the question each importance measure answers and explain the convenience and splitting-opportunity limitation of impurity-based importance.
- Conditional permutation importance. Explain why a low permutation-importance score can reflect redundant information rather than an unhelpful predictor.
- Predictive importance versus causality. Distinguish a forest’s reliance on a feature for prediction from the effect of intervening to change that feature.
- Two paths to generalization. Explain how SVMs constrain a single model while random forests stabilize predictions across many variable trees, and why neither approach is universally superior.
- Organized diversity and reliability. Explain how bagging and random feature selection turn unstable individual trees into a more reliable forest, and why tree strength still matters.
What it builds on
- Random forests (chapter 5) · Introduces random forests in the context of data partitioning.
- Why support vector machines generalize (chapter 8) · Discusses why support vector machines generalize.
- Decision trees (chapter 5) · The earlier introduction to decision trees as data partitions.
- Parallel margins and support vectors (chapter 8) · Introduces the margin associated with a support vector machine.
- The penalty parameter and soft-margin optimization (chapter 8) · Introduces the SVM optimization trade-off involving a wider boundary.
- Selecting the penalty parameter through cross-validation (chapter 8) · Cross-validation is used to assess model choices.
- Constraints, margin width, and optimization (chapter 8) · Introduces the SVM’s geometric margin.
- From explicit transformations to the kernel trick (chapter 8) · Introduces nonlinear kernels that can complicate interpretation of an SVM decision function.
Section by section
Introduction
This introduction explains why unstable decision trees can benefit from an ensemble and contrasts Breiman’s approach to generalization with Vapnik’s.
- Single tree instability: Explain why a decision tree can fit its training data yet fail to generalize reliably.
Why it matters: This instability is the problem Breiman’s ensemble approach addresses. - Diversity through aggregation: Explain Breiman’s central idea of combining deliberately different trees rather than relying on one tree.
Why it matters: It captures how a weakness of individual trees can become a strength of the ensemble. - Contrasting generalization strategies: Distinguish Vapnik’s and Breiman’s approaches to making predictions generalize.
Why it matters: The comparison clarifies what is distinctive about Breiman’s route to generalization.
Key terms: decision trees, unstable, generalize, randomness, diversity, aggregation, ensemble, generalization, support vector machines
9.1 The strengths and limitations of single decision trees
This section explains how a single decision tree builds and makes predictions, why it is useful, and why its flexibility can make it unreliable on new data.
- Tree splits and predictions: Trace how a decision tree recursively splits training observations and predicts for a new observation. Distinguish the split criteria and leaf predictions used for classification and regression.
Why it matters: The split-and-leaf process is the basis for understanding both individual trees and the forests built from them. - Tree flexibility and interpretability: Explain why single decision trees are useful for modeling and communicating predictions.
Why it matters: These strengths explain why an individual tree remains appealing even when its predictions have limitations. - Single-tree instability and overfitting: Explain how early splits and increasing depth can make a single tree unreliable on new observations.
Why it matters: This limitation motivates looking beyond a single tree for more stable predictions.
Key terms: root, split, impurity, entropy, information gain, node, conditional decisions, nonlinear relationships, interactions, plotted tree
9.1.1 Instability and sensitivity to early splits
This section explains why greedy split choices make a single decision tree sensitive to small changes in its training data, especially near the root.
- Greedy split selection: Explain that a tree chooses the best immediate split at each node for the observations there, without looking ahead or revisiting earlier choices.
Why it matters: Local choices make trees practical to train but help explain why their structures can be fragile. - Early split instability: Trace how a small sample change can alter a near-tied split, particularly near the root, and thereby change the data and split choices in downstream nodes.
Why it matters: It explains why trees fitted to nearly identical samples may end up with very different structures. - Fragile single-tree interpretation: Describe why a well-fitted, interpretable tree may still fail to retain its structure across samples or reflect patterns that persist in new observations.
Why it matters: An explanation of one fitted tree should not automatically be treated as a stable finding about future data.
Key terms: greedily, candidate splits, immediate improvement, root splits, features, thresholds, branches, instability, training sample, new observations
9.1.2 Depth, overfitting, and variance
This section explains how deeper decision trees gain expressive power but become more prone to overfitting and high variance.
- Depth and local partitions: Explain how successive splits create smaller groups and let a deep tree capture relationships a shallow tree may miss.
Why it matters: The benefit of depth helps explain why simply making every tree shallow has a cost. - Depth and overfitting: Explain why splits based on very small groups can fit training noise and make training performance misleading about performance on new data.
Why it matters: An excellent fit to observed data is not, by itself, evidence that the tree will predict new cases well. - Variance and depth trade-off: Describe why deep trees have high variance and explain the trade-off when their depth is restricted.
Why it matters: The issue is a balance between a tree that captures complex patterns and one whose predictions remain stable.
Key terms: split, node, depth, expressive power, noise, overfitting, training performance, generalization, high-variance models, variance
9.1.3 Why pruning is not enough
The section explains why pruning improves a decision tree but cannot remove its sample-dependent instability, motivating the use of varied trees whose predictions are combined.
- Tree constraints and pruning: Describe how limiting splits during training or removing branches afterward can improve a tree’s generalization.
Why it matters: This explains what pruning accomplishes before considering what it leaves unresolved. - Limits of one pruned tree: Explain why pruning cannot eliminate instability and what flexibility a tree gives up when it is restricted.
Why it matters: It identifies why improving one tree is not the same as solving the single-tree problem. - Stability through varied trees: Explain why combining predictions from varied trees offers an alternative to making one tree stable, and state the basic bagging approach introduced here.
Why it matters: It shows why stability can come from a collection of trees rather than from heavily restricting one tree.
Key terms: maximum depth, minimum number of observations, minimum improvement, pruned, generalization, pruned tree, instability, single sample, variance, nonlinear relationships
9.2 Bagging: randomness in the observations
The section explains how bagging turns the sensitivity of unstable models, especially decision trees, into more stable predictions by combining models trained on varied observation samples.
- Bagging and tree stability: Explain why decision trees are well suited to bagging and how combining trees trained on varied observation samples stabilizes predictions without making each tree less expressive.
Why it matters: It shows why variation that hurts a single tree can help when many trees are combined. - Bagging and forest origins: Distinguish bagging from the later random forest idea at the level described here: bagging varies observations, while random forests retain that randomness and add randomness involving the features considered during tree building.
Why it matters: It prevents treating bagging and random forests as the same invention.
Key terms: bagging, unstable learning methods, voting, averaging, random forests, randomness in observations, randomness in features
9.2.1 How bagging creates different training sets
This section explains how bootstrap sampling gives bagged decision trees different training observations, why some observations are omitted, and how those differences can produce varied trees whose predictions are combined.
- Bootstrap samples within bagging: Describe how bagging begins with bootstrap sampling: repeatedly draw observations with replacement to make training samples of the original size, then train a model on each sample and aggregate predictions.
Why it matters: The distinction explains where bagging’s variation comes from before predictions are combined. - Omitted and repeated observations: Explain why a bootstrap sample can repeat some observations and omit others, and state the approximate proportions that appear at least once or are omitted in a large sample.
Why it matters: The omitted observations make each training set different and later provide data for evaluating trees. - Sampling effects on tree structure: Explain how training separate trees on bootstrap samples can change their splits and structures even though the tree-training method and available features stay the same.
Why it matters: It connects resampling to the diversity of trees that makes aggregation useful.
Key terms: bagging, bootstrap sampling, sampling with replacement, aggregation, out-of-bag observations, bootstrap sample, decision tree, candidate splits
9.2.2 Aggregation: turning variation into stability
This section explains how bagging combines tree predictions and why aggregation can make an ensemble more stable than its individual trees.
- Classification votes and regression averages: Describe how bagging produces one prediction from trees trained on bootstrap samples: it selects the most-voted class for classification and averages numerical predictions for regression.
Why it matters: The way a forest combines its trees depends on what it is predicting. - Ensemble stability through aggregation: Explain why combining predictions reduces variance even though individual bootstrap-trained trees remain sensitive to their samples.
Why it matters: This explains how a collection of unstable trees can yield a steadier prediction. - Deep trees and correlated errors: Distinguish bagging’s route to stability from pruning, and explain why strongly correlated tree predictions limit the benefit of aggregation.
Why it matters: Aggregation helps preserve what deep trees can capture, but it works best when their errors differ.
Key terms: bagging, classification, regression, votes, averages, aggregation, bootstrap sample, ensemble, variance, sample-specific effects
9.2.3 What bagging fixes—and what it does not
This section explains how bagging stabilizes decision trees, why correlated trees limit its gains, and how random forests add diversity beyond bagging.
- Bagging and variance reduction: Explain how bootstrap samples change the trees that are trained and how combining their predictions reduces the influence of any one sample or tree.
Why it matters: It explains why combining unstable trees can produce a more stable prediction. - Correlated errors limit aggregation: Explain why bootstrap sampling alone may leave trees too similar and why adding more such trees offers diminishing benefit.
Why it matters: The number of trees is not enough; their errors must differ for aggregation to help much. - Random forests add feature randomness: Describe the additional source of diversity random forests introduce beyond bootstrap sampling and why it addresses a limit of bagging.
Why it matters: It shows how random forests respond to bagging's problem of trees making similar errors. - Bagging versus boosting: Distinguish how bagging and boosting create and combine models, and state their different routes to improving an ensemble.
Why it matters: Both are ensembles, but they create diversity and improve predictions in different ways.
Key terms: bagging, bootstrap sample, high variance, aggregation, bootstrap sampling, correlated errors, diversity, early splits, random forests, randomly selected subset
9.3 Random forests: randomness in the features
The section explains how random forests add random feature selection to bagging so that trees make less similar errors and aggregation works better.
- Bagging’s remaining similarity problem: Explain why bootstrap sampling alone may leave decision trees too similar: strong predictors can dominate early splits across trees, leading to similar decisions and errors.
Why it matters: It explains why bagging benefits from another source of randomness. - Two sources of forest randomness: Describe how a random forest combines bootstrap samples with a newly selected random subset of features at each node, chooses the best split from that subset, and aggregates predictions across trees.
Why it matters: It distinguishes a random forest from bagging trees using bootstrap samples alone. - Diversity and aggregation tradeoff: Explain why withholding strong predictors at some splits can make individual trees slightly less accurate yet improve the aggregated forest by making tree structures and errors less correlated.
Why it matters: It explains why choosing the strongest available predictor at every split is not necessarily best for the ensemble.
Key terms: bagging, bootstrap sampling, predictors, random feature selection, split, aggregation, dominant predictors, diversity, correlated errors
9.3.1 Random feature selection at each split
This section explains how random forests restrict the features considered at each split while retaining ordinary tree splitting and adding diversity beyond bagging.
- Fresh feature subsets at splits: Explain how a random forest draws a new subset of m out of p features at every node and chooses the best split available within that subset. Describe how the rest of the decision-tree process remains the same.
Why it matters: This is the defining change from an ordinary or bagged tree to a tree in a random forest. - Two sources of tree variation: Distinguish bootstrap sampling of observations from random feature selection, and explain why adding feature randomness can make the trees follow different paths rather than repeatedly favoring dominant predictors.
Why it matters: The distinction explains what feature randomness adds to bagging and why the forest's trees can differ.
Key terms: input features, random subset, node, split, m, p, bootstrap sampling, random feature selection, with replacement, without replacement
9.3.2 Creating diversity by limiting choice
This section explains how limiting feature choices at each split makes random-forest trees less alike, and why that diversity must be balanced against each tree’s predictive strength.
- Feature limits and tree diversity: Trace how withholding a dominant predictor at some splits leads trees to choose other useful predictors and develop different structures, without permanently removing the dominant predictor.
Why it matters: It explains why limiting a tree’s choices can create more diversity than changing its training sample alone. - Diversity and error correlation: Explain why structurally different trees are less likely to make the same errors and how that makes voting or averaging more effective at reducing variance.
Why it matters: Diversity helps the forest because its trees are less likely to repeat the same mistakes. - Feature subset size trade-off: Explain why feature subsets that are too small can weaken individual trees, while subsets that are too large can leave trees overly similar.
Why it matters: The strongest possible individual trees do not necessarily produce the strongest forest.
Key terms: bagging, dominant predictor, random feature selection, feature subset, early splits, structural diversity, correlated, voting, averaging, variance
9.3.3 The complete random forest algorithm
This section explains how random forests train varied decision trees using bootstrap samples and random feature subsets, then combine their predictions.
- Randomized tree construction: Trace how a random forest trains many trees: draw a new bootstrap sample for each tree, restrict the candidate features separately at each split, and grow trees using ordinary recursive splitting.
Why it matters: It shows where randomness enters the algorithm without mistaking the individual trees for a different kind of learner. - Forest prediction aggregation: Describe how the trained trees produce one forest prediction for classification and for regression.
Why it matters: Aggregation is the step that turns many separately trained trees into one model prediction.
Key terms: bootstrap sample, random feature selection, greedy recursive splitting, impurity or loss criterion, aggregation, classification, regression, votes, averaged
9.4 Why random forests generalize
This section explains why random forests generalize: they combine useful individual trees whose errors differ, then stabilize predictions through aggregation.
- Tree strength and diversity: Explain why reliable generalization requires trees that predict better than chance without repeatedly making the same errors, and how random feature selection helps balance those requirements.
Why it matters: It explains why making trees different helps only if they remain useful predictors. - Forest margin and variance: Describe what the forest margin indicates and distinguish the forest’s way of reducing variance from an SVM’s.
Why it matters: It connects the forest’s collective confidence to its distinct approach to generalization. - Adding trees and stability: Explain what changes as more trees are added and why tree count alone is not the decisive consideration.
Why it matters: It prevents mistaking a larger forest for an automatically better one.
Key terms: strength, low correlation, random feature selection, forest margin, geometric margin, variance, diversification, aggregation, limiting level of generalization error, individual strength
9.4.1 Classification margins: measuring collective confidence
This section defines Breiman’s classification margin from tree votes and shows how margin signs and distributions help evaluate a random forest without changing how it is trained.
- Classification margin from votes: Explain how to calculate a classification margin for an observation with a known class and interpret its sign and size.
Why it matters: A margin distinguishes a decisive correct vote from one that could easily become a tie or error. - Margin distribution and error: Explain how margins across labeled observations relate to classification error and what their distribution reveals beyond accuracy.
Why it matters: Forests with equal accuracy can differ in how narrowly their correct predictions are supported. - Margins as forest diagnostics: Distinguish a random forest’s use of margins from an SVM’s, and describe how out-of-bag margins can inform a forest comparison.
Why it matters: Margins help assess a forest without being mistaken for an objective its training directly maximizes. - Formal margin applicability limits: Distinguish Breiman’s formal margin from a vote gap for an unlabeled case and from measures used for regression forests.
Why it matters: These limits prevent a vote gap or the spread of numerical predictions from being misreported as a formal margin.
Key terms: Breiman’s margin, known class, vote share, generalization error, margin distribution, classification error, out-of-bag margins, geometric margin, candidate features, formal margin
9.4.2 Balancing tree strength and correlation
The section explains why random forests generalize best when their trees make useful predictions without making the same errors, and how feature selection affects that balance.
- Classification strength and correlation: Explain how forest strength is defined from classification margins and why a forest needs both useful trees and differences in their errors.
Why it matters: Many trees help only when their votes are informative rather than repetitions of the same mistakes. - Classification error bound: Describe what Breiman’s classification-error bound says about strength and correlation, and distinguish a bound from an exact error prediction.
Why it matters: The bound makes clear why neither tree strength nor diversity should be judged alone. - Regression averaging and shared errors: Explain why averaging regression trees can reduce variance when their errors are not closely correlated, and why diversity cannot remove shared bias.
Why it matters: It explains both the benefit and the limit of averaging more regression trees. - Candidate feature trade-off: Explain how the number of candidate features at each split trades individual tree quality against similarity among trees, and how forests can be compared to choose a balance.
Why it matters: Feature sampling is a practical way to seek useful trees whose errors are not too alike.
Key terms: forest margin, strength, correlation, generalization error, upper bound, average correlation, regression forest, prediction variance, systematic bias, mean squared error
9.4.3 Why adding trees stabilizes the forest
This section explains why adding randomized trees stabilizes a forest’s predictions without guaranteeing that those predictions are accurate.
- Tree Depth Versus Tree Count: Distinguish the effect of deepening one decision tree from adding another tree to a random forest. Explain that an added tree is separately randomized and equally weighted rather than making existing trees more complex.
Why it matters: Adding trees and adding depth change a model in fundamentally different ways. - Convergence of Forest Predictions: Explain how predictions stabilize as the number of randomized trees grows: regression averages approach an expected prediction, while classification vote proportions approach expected vote shares. Describe why error tends to plateau rather than systematically worsen merely because more trees are added.
Why it matters: A larger forest mainly makes its collective prediction steadier, not endlessly better. - Limits of a Stable Forest: Explain why convergence does not guarantee accurate predictions or prevent a forest from performing poorly. Relate the quality of the stable result to tree strength and correlation, and state why judging it requires observations not used to train the relevant trees.
Why it matters: A prediction can be reliably stable and still be wrong.
Key terms: depth, bootstrap sample, randomly selected feature subsets, limiting value, expected prediction, expected vote share, out-of-bag error, tree strength, correlation, finite ensemble
9.5 Out-of-bag evaluation
The section explains how bootstrap sampling lets a random forest evaluate its predictions using out-of-bag observations, and where that evaluation is useful or limited.
- Out-of-bag prediction construction: Trace how a training observation receives an out-of-bag prediction from only the trees whose bootstrap samples omitted it, using votes for classification or an average for regression.
Why it matters: This explains how the forest obtains predictions from trees that did not train on the cases they are judging. - Internal validation and error measures: Explain why out-of-bag predictions provide internal validation and describe how they are compared with known outcomes for classification or regression.
Why it matters: Out-of-bag evaluation estimates predictive performance using work already done while fitting the forest. - Cross-validation comparison and scope: Distinguish out-of-bag evaluation from k-fold cross-validation in how they withhold observations and in the kinds of validation designs they support.
Why it matters: The comparison clarifies when convenient built-in OOB evaluation may not replace a deliberately designed validation procedure. - Out-of-bag use and caution: Describe practical uses of out-of-bag results and explain why repeated tuning on those results calls for caution and may leave a role for an independent test set.
Why it matters: A convenient internal estimate is useful for tuning, but it should not automatically be treated as an unbiased final verdict.
Key terms: bootstrap sample, out-of-bag observations, out-of-bag prediction, internal validation, out-of-bag error rate, mean squared error, mean absolute error, k-fold cross-validation, fixed folds, overlapping validation sets
9.6 Variable importance: recovering interpretation
The section explains why feature importance is harder to interpret in a random forest than in a single tree and introduces ways to measure it across the ensemble.
- Single-tree feature interpretation: Explain how a tree’s plotted splits help show which features influence predictions, and why their positions alone do not give a complete numerical ranking of importance.
Why it matters: It distinguishes what a tree diagram reveals from what an importance score must measure. - Forest-wide feature importance: Explain why interpreting a random forest requires ensemble-level measures rather than tracing one tree, and describe the two broad approaches introduced here, including the role of out-of-bag observations.
Why it matters: It shows how a forest can yield feature-level insight even though it cannot be read as one tree.
Key terms: decision tree, splits, importance scores, random forest, out-of-bag observations, feature, variable importance
9.6.1 Measuring variable importance
This section explains how random forests measure variable importance through out-of-bag permutation and accumulated split improvements, and how to interpret the resulting scores.
- Permutation importance calculation: Trace how permutation importance compares baseline out-of-bag error with error after shuffling one feature, without retraining the forest.
Why it matters: It shows whether the forest’s predictions depend on the information a feature carries. - Permutation score interpretation: Explain what positive, near-zero, and negative permutation-importance scores indicate, including why zero or negative scores need cautious interpretation.
Why it matters: The size and sign of a score describe a change in measured error, not a definitive verdict on a feature’s usefulness. - Impurity reduction accounting: Describe how impurity-based importance credits features for weighted split improvements across a forest and what a normalized score represents.
Why it matters: It distinguishes meaningful split improvements from simply counting how often a feature is used. - Importance measures and limitations: Distinguish the question each importance measure answers and explain the convenience and splitting-opportunity limitation of impurity-based importance.
Why it matters: Choosing and reading an importance measure requires knowing whether it reflects predictive dependence or credited training-time splits.
Key terms: permutation importance, out-of-bag performance, baseline error, permutation-importance score, classification error, mean squared error, impurity-based importance, weighted reduction, Gini impurity, entropy
9.6.2 interpreting importance responsibly
This section explains what random-forest variable-importance scores can reveal and why redundancy, scoring methods, causality, and global aggregation limit their interpretation.
- Conditional permutation importance: Explain why a low permutation-importance score can reflect redundant information rather than an unhelpful predictor.
Why it matters: A low score need not mean the underlying information is useless. - Impurity score biases: Describe why impurity-based importance can favor features with many candidate splits or sample-specific patterns.
Why it matters: A high impurity-based rank can partly reflect opportunities to split rather than greater predictive contribution. - Predictive importance versus causality: Distinguish a forest’s reliance on a feature for prediction from the effect of intervening to change that feature.
Why it matters: Predictive rankings alone cannot tell you what action would change an outcome. - Global versus individual explanations: Explain why a global importance ranking cannot by itself explain an individual prediction.
Why it matters: An overall ranking can hide features that matter greatly to one observation or a small subgroup. - Forest-level transparency: Describe how variable-importance rankings provide interpretation when a forest lacks a single traceable decision path.
Why it matters: Losing a single visible decision path does not make the ensemble entirely uninterpretable.
Key terms: permutation importance, correlated predictors, redundancy, impurity-based importance, splitting points, sample-specific patterns, variable importance, causality, proxy, intervention
9.7 Random forests versus support vector machines
This section contrasts how support vector machines and random forests achieve generalization, use margins, and provide evidence for interpreting predictions.
- Two paths to generalization: Explain how SVMs constrain a single model while random forests stabilize predictions across many variable trees, and why neither approach is universally superior.
Why it matters: The comparison shows that reliable predictions can come from either constraining one model or combining many variable ones. - Distinct meanings of margin: Distinguish the geometric margin used in SVM training from the margin calculated from random-forest votes.
Why it matters: The shared word “margin” refers to different roles in training and evaluating these methods. - Different forms of interpretive evidence: Describe what support vectors reveal about an SVM and what out-of-bag evaluation and variable importance reveal about a random forest, including their limits of transparency.
Why it matters: Interpretation means looking for different kinds of evidence rather than expecting either model to read like a small tree.
Key terms: generalization, variance, soft-margin SVM, random forests, aggregation, margin, geometric margin, classification violations, random forest margin, support vectors
9.8 The enduring legacy of random forests
The section explains why random forests endure: they organize unstable trees into a reliable ensemble, provide evidence about performance, and call for a different kind of interpretation.
- Organized diversity and reliability: Explain how bagging and random feature selection turn unstable individual trees into a more reliable forest, and why tree strength still matters.
Why it matters: The forest’s reliability comes from organizing imperfect trees, not requiring each tree to be reliable on its own. - Practical utility and evaluation: Describe why random forests remain useful in practice and how the section connects their diagnostic tools to judging predictions on new data.
Why it matters: A useful model should offer evidence about how it may perform beyond its training observations. - Interpretation beyond individual trees: Distinguish the direct transparency of a single tree from the way the section says to understand a forest.
Why it matters: Losing a single visible decision path does not mean losing every way to understand or assess the model.
Key terms: bagging, bootstrap samples, random feature selection, tree strength, correlation, aggregation, structured or tabular data, out-of-bag observations, variable importance, predictive performance
9.9 Summary
Where this is used later
- Breiman: stabilizing predictions through randomness and aggregation (chapter 10) · The random forest algorithm.
- What depth, width, and learned representations contribute (chapter 11) · Discusses stable predictions and generalization in random forests.
Concepts explained in this chapter
- random forestA random forest is an ensemble model that combines predictions from multiple decision trees trained with randomized sampling or feature selection.
- baggingBagging, or bootstrap aggregation, is an ensemble technique that trains models on bootstrap samples and combines their predictions to reduce variability.
- random feature selectionRandom feature selection restricts the features considered at a decision-tree split to a randomly chosen subset.
- data aggregationAggregation is the combination of multiple observations, models, or outputs into a consolidated result.
- bootstrap samplingA resampling technique that draws observations with replacement from an original dataset.
- feature importance scoreVariable importance is a quantitative assessment of how much a predictor contributes to a model's predictions or reduction of uncertainty.
- varianceVariance is the model's sensitivity to changes in the training sample, reflected in how its behavior varies across samples.
- decision treeA decision tree is a hierarchical predictive model that recursively partitions data using feature-based rules.
- correlationCorrelation measures the strength and direction of association between quantities.
- impurity-based importanceImpurity-based importance is a measure that aggregates the impurity reductions attributed to a feature across a model's tree splits.
- permutation feature importancePermutation importance measures a feature's contribution to a model by assessing the effect of permuting its values on predictive performance.
- marginA boosting margin measures the combined classifier's support for the correct class relative to a competing class.
- generalizationGeneralization is the ability of a learned model to perform effectively on observations not used during training.
- out-of-bag evaluationOut-of-bag evaluation estimates a model’s predictive performance using observations omitted from the samples used to train the models making those predictions.
- pruningPruning is a technique that reduces a decision tree by removing branches to limit its size or improve its predictive behavior.
- stabilityStability is the persistence of patterns and relationships in data under comparable conditions or over time.
- support vector machineA support vector machine is a supervised learning model that finds a class-separating boundary, typically by maximizing the margin.
- ensembleEnsemble learning combines multiple predictive models to improve predictions.
- forest marginA forest margin is the difference between a forest's voting support for an observation's correct class and its strongest competing class.
- strengthTree strength measures the predictive ability of individual trees in a forest.
- model diversityEnsemble diversity is the extent to which constituent models differ in their structures, predictions, or errors.
- splitA decision-tree split partitions observations at a node according to a feature-based rule.
- margin distributionA margin distribution is the distribution of classification-margin values across labeled observations.
- error rateAn error rate is the proportion of predictions or classifications that are incorrect.
- overfittingOverfitting occurs when a model fits its training observations too closely without capturing relationships that remain useful beyond them.
- out-of-bag observationAn out-of-bag observation is a training observation omitted from a particular bootstrap sample and evaluated using trees that did not include it.
- bootstrap sampleA dataset formed by drawing observations with replacement from an original dataset.
- causationCausality, or causation, is a relationship in which a change in one factor brings about a change in another.
- out-of-bag errorOut-of-bag error estimates prediction error using observations omitted from the bootstrap samples used to train the relevant trees.
- boostingBoosting is an ensemble technique that builds models sequentially, with later models focusing on cases earlier models handled poorly.
- soft-margin extensionSoft-margin optimization finds a separating boundary by balancing margin width against penalties for margin violations.
- information gainInformation gain is the reduction in uncertainty produced by a split or observation, commonly measured using entropy.
- node impurityImpurity is a measure of how mixed the outcomes or classes are within a decision-tree node.
- expressive powerThe range and complexity of relationships that a model can represent.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
