Tree ensemble

As explained in Ensemble Methods for Machine Learning

A tree-based ensemble is an ensemble model whose component predictors are decision trees or tree-derived models.

A tree ensemble combines decision trees or tree-derived models into one predictor. Tree-based ensembles are especially useful when the goal is strong predictive performance or competitive analytics. section 1.2 section 9.2.3

Common tree ensembles include:

• Random forests, which use collections of decision trees. section 9.2.3
• XGBoost and gradient boosting, which also rank among the strongest methods in benchmark comparisons. section 1.2
• Extra Trees and AdaBoost, which also outperform many other evaluated methods. section 1.2

Comparing ensemble decisions

On 165 benchmark data sets, XGBoost beat multinomial naïve Bayes on 157 data sets, while gradient boosting and XGBoost performed similarly on most remaining data sets. For interpretation, random forest and XGBoost feature importances highlighted employment, interest-rate, and consumer-confidence indicators when predicting fixed-term deposit subscriptions. section 1.2 section 9.2.3

Figure 1.2 Which machine-learning algorithm should I use for my data set? The performance of several different machine-learning algorithms, relative to each other on 165 benchmark data sets, is shown here. The final trained models are ranked (top-to-bottom, left-to-right) based on their performance on all benchmark data sets in relation to all other methods. In their evaluation, Olson et al. consider two methods to have the same performance on a data set if their prediction accuracies are within 1% of each other. This figure was reproduced using the codebase and comprehensive experimental results compiled by the authors into a publicly available GitHub repository (https://github.com/rhiever/sklearn-benchmarks) and includes the authors’ evaluation of XGBoost as well.

Compare the relative performance of tree ensembles and other machine-learning algorithms across 165 benchmark data sets. section 1.2

How Ensemble Methods for Machine Learning covers it

How deep it goes

Defines tree ensembles through their decision-tree base estimators and shows feature-importance interpretation, but does not detail training mechanics.

Examples

Uses 165 benchmark data sets and a deposit-subscription classifier whose important features indicate employment, interest rates, and consumer confidence.

What you can do after

Evaluate ensemble performance across data sets and interpret overall model behavior through feature importances.

How it connects

Connects decision trees with random forests, XGBoost, gradient boosting, Extra Trees, AdaBoost, and global explainability.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Related concepts

• decision treehas part · A decision tree is a branching structure that represents sequential tests or choices, with paths leading to outcomes, predictions, or actions.
• random forestincludes · A random forest is an ensemble model that combines predictions from multiple decision trees built with randomized data samples, feature selections, or both.
• ensemblekind of · An ensemble is a technique that combines or considers outputs from multiple models or component methods collectively to produce or assess a joint result.
• irrelevant featurecontrasts with · An irrelevant feature is an input attribute that provides little or no useful information for predicting a target.
• XGBoostincludes · XGBoost is an open-source software library for efficient gradient-boosted decision-tree algorithms used in supervised machine learning.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.