Decision tree

As explained in Ensemble Methods for Machine Learning

A decision tree is a predictive model that recursively partitions data according to feature-based decisions, producing predictions at the resulting leaves.

A decision tree predicts by asking a sequence of feature-based questions. Each decision node sends an example down one branch; the leaf reached returns a class label for classification or a numeric value for regression. The resulting paths partition feature space into nonoverlapping regions, making the model’s logic directly inspectable. section 1.3.1 section 9.1.2

Figure 1.3 Decision trees partition the feature space into axis-parallel rectangles. When used for classification, the tree checks for conditions on the features in the decision nodes, funneling the example to the left or right after each test. Ultimately, the example filters down to a leaf node, which will give its classification label. The partition of the feature space according to this decision tree is shown on the left.

The feature-space view shows how successive threshold tests carve inputs into axis-parallel regions, with a leaf prediction assigned to each region. section 1.3.1

Tree depth controls complexity. A depth-1 tree, or decision stump, has one test and two leaves, so it represents only simple functions. Adding nodes creates richer feature combinations and nonlinear behavior, but a very deep tree can fit training data nearly perfectly while generalizing poorly to new data. section 1.3.1 section 7.1.5

A practical workflow is:

• Start with labeled training data and reserve test data for evaluation. section 1.3
• Train trees across a range of maximum depths rather than assuming that a more complex tree is better. section 1.3.1
• Compare training and test R² for regression; a widening gap signals overfitting. section 1.3.1
• Inspect the learned splits and paths to identify the features and rules driving predictions. section 9.1.2

Nonlinear regression with a bounded-depth tree

For noisy samples from a one-variable nonlinear function, a DecisionTreeRegressor with max_depth=5 is trained on a training split and evaluated with mean squared and mean absolute error on held-out data. The tree is one of two nonlinear regression approaches compared, alongside a neural network regressor. section 7.1.5

Trees are useful when prediction logic must be understandable. On Iris classification, root-to-leaf paths can be rewritten as ordinary conditional rules; a trained tree uses petal width and petal length to separate three species. Such sparse, sequential rules make a tree a glass-box model rather than a black-box predictor. section 9.1.2

Figure 9.3 Decision tree learned on the Iris data set for classification of irises into three species: Iris setosa, Iris versicolour, and Iris virginica. The standard convention for splits is followed here: if the split function evaluates to true, we proceed to the right branch; if it evaluates to false, the left branch.

The rendered Iris tree makes each split, branch choice, and leaf-level class prediction visible as a readable decision process. section 9.1.2

How Ensemble Methods for Machine Learning covers it

How deep it goes

Explains nodes, leaves, depth, nonlinear growth, evaluation curves, and rule interpretation without deriving split optimization.

Examples

Uses Friedman-1 regression, a noisy nonlinear function, and Iris species classification with visualized rules.

What it assumes

Reinforces labeled supervised-learning tasks and distinguishes binary, multiclass, and continuous targets before tree use.

What you can do after

Apply it via scikit-learn to train bounded-depth regressors or classifiers, evaluate held-out performance, and inspect learned rules.

How it connects

Connects decision trees to supervised learning, regression, classification, overfitting, neural networks, and ensemble methods.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningAn algorithm that recursively splits labeled data using feature tests and stopping conditions to construct a decision tree.
• 

  liveBookmanning.com
  Statistics Every Programmer NeedsA decision tree is a graphical decision-analysis model that represents choices, uncertain events, possible outcomes, probabilities, and payoffs as branching paths.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA decision tree is a predictive model that recursively partitions inputs using feature-based split rules and produces outcomes at terminal leaves.
• 

  liveBookmanning.com
  Statistics Slam DunkA decision tree is a predictive model that recursively partitions data using feature-based decision rules.
• 

  liveBookmanning.com
  Outlier Detection in PythonA decision tree is a hierarchical predictive model that recursively partitions observations using feature-based decision rules.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionA decision tree is a predictive model that recursively applies feature-based tests to partition examples and produce a classification or regression outcome.
• 

  liveBookmanning.com
  Data Without LabelsA tree-structured predictive model that recursively splits data into nodes using decision rules.
• 

  liveBookmanning.com
  Inside AIA decision tree is a hierarchical predictive model that recursively splits data according to attribute-based questions to reach decisions or predictions.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthA decision tree is a hierarchical predictive model that makes decisions by applying feature-based tests at internal nodes.
• 

  liveBookmanning.com
  Machine Learning Platform EngineeringA decision tree is a predictive model that recursively partitions input data using feature-based decision rules.
• 

  liveBookmanning.com
  Demand Forecasting Best PracticesA decision tree is a machine-learning model that makes predictions by applying a sequence of feature-based questions and following the resulting branches to a terminal prediction.
• 

  liveBookmanning.com
  Data-Driven CybersecurityA predictive model that recursively partitions data through feature-based decision rules arranged in a tree structure.

Compare how 13 books explain decision tree →

Related concepts

• leaf nodehas part · A leaf node is a node in a tree or hierarchical structure that has no child nodes and cannot be further subdivided.
• random forestpart of · A random forest is an ensemble model that combines predictions from multiple decision trees built with randomized data samples, feature selections, or both.
• language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• decision nodehas part · A decision node is a tree node where a decision, test, or action selection determines which successor branch is followed.
• decision stumpincludes · A decision stump is a decision tree of depth one with a single decision node and prediction leaves.
• XGBoostused by · XGBoost is an open-source software library for efficient gradient-boosted decision-tree algorithms used in supervised machine learning.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.