A decision tree predicts by repeatedly splitting a dataset with feature-based rules. Start at the root, follow branches that satisfy each condition, and end at a leaf that supplies the prediction. For classification, splits seek subsets dominated by one class; for regression, they seek subsets with less target-value variability. Trees work naturally with tabular data and need relatively little preprocessing. section 5.1
Figure 5.1 The key elements, such as roots, branches, and leaves, constituting a decision tree classifying animals based on the number of legs and eyes
Use the animal classifier to read a tree as a sequence of plain-language questions: each node tests a feature, each branch routes a case, and each leaf decides an outcome. section 5.1
A tree searches available features and observed values for useful splits. Classification commonly evaluates target homogeneity with entropy, information gain, or Gini impurity. Binary true-or-false splits are typical, although some variants make multiple branches. Repeated splitting lets a tree capture nonlinear relationships and feature interactions as if-then-else rules. section 5.1 section 6.1.4
Classifying Airbnb listings
A practical classifier can predict whether an Airbnb New York listing is above or below the median price. Build a pipeline that preprocesses categorical and numeric columns, fits scikit-learn’s DecisionTreeClassifier, and evaluates accuracy with shuffled five-fold cross-validation. This grounds tree choices in a realistic tabular-data prediction task rather than a diagram alone. section 5.1
Control tree complexity to balance fit and generalization:
- Limit growth or prune a fully grown tree; a smaller tree represents a simpler decision problem. section 5.1
- Set max_depth to cap levels of splitting, and use min_samples_split or min_samples_leaf to require enough data before splitting or forming leaves. section 5.2.1 section 5.3
- Treat deeper, more complex trees as an overfitting risk, even when they model more complicated data patterns. section 5.3
A single tree is also the building block for tree ensembles. Random forests combine trees through averaging, while gradient boosting chains trees to improve predictions sequentially. Gradient-boosted decision trees are emphasized as a strong approach for tabular problems, including implementations such as XGBoost and LightGBM. In boosting, the individual trees are regression trees, even for classification. section 5.0 section 5.2.1
Tree-based rules can handle nonlinearities and interactions automatically, but their binary partitions are rough approximations. Noise can make parts of the estimated space inaccurate, and neural networks may form smoother approximations when complex relationships and enough examples are available. When important nonlinearities or interactions are known, transformations, binning, or precomputed interactions can reduce the splits a gradient-boosted tree needs. section 6.1.4
Inspecting a boosted tree
A fitted XGBoost or LightGBM model can expose its tree structure. Follow a path with depth-first traversal and inspect the feature, branch direction, threshold, gain, and sample count at each node; these details allow a tree’s result to be replicated. Package plotting utilities provide a graphical view of the same structure. section 6.4.2
