Extremely randomized trees, commonly called Extra Trees, form a parallel ensemble from many decision trees. Their purpose is the same as other tree ensembles: combine diverse base learners so that the aggregate prediction is more stable than a single tree. Extra Trees creates that diversity by adding randomness while each tree grows, making it a practical choice when training efficient tree ensembles on high-dimensional data with many continuous features. section 2.4 section 2.4.3 section 2.6
The key idea is to randomize both parts of a tree split. At a node, a tree asks a question such as “is feature fk less than a threshold?” A standard decision tree searches all features and their candidate values for the best split. A randomized decision tree first considers only a random feature subset, then finds the best threshold among that feature’s values. An extremely randomized tree also starts with a random feature subset, but uses a random splitting threshold. The chosen feature is therefore evaluated with this deliberately random threshold rather than by exhaustively searching thresholds. section 2.4.3 section 7.2.1
Compared with closely related tree learners, the source of randomness changes as follows:
- Decision tree: searches all features and candidate feature values for the best split. section 2.4.3
- Randomized decision tree: searches a random subset of features, then finds the best threshold for the selected feature. section 2.4.3
- Extremely randomized tree: uses a random feature subset and a random split threshold, producing still more randomized base learners. section 2.4.3
- Extra Trees ensemble: aggregates many extremely randomized trees, typically trained directly on the original data rather than bootstrap samples. section 2.4.3 section 2.6
Avoiding bootstrap sampling is an important practical distinction from random forests. Random forests use bootstrap samples and randomized decision trees; Extra Trees relies on extreme split randomization to generate diversity and can therefore train its ensemble directly from the original data set. This additional randomization can improve training efficiency and reduce computational requirements while increasing diversity across the trees. section 7.2.1 section 2.4.3 section 2.6
Classification: breast cancer diagnosis
For a diagnostic classification task, Extra Trees can be compared with bagging and random forests after preprocessing the Wisconsin Diagnostic Breast Cancer data. Evaluate training and held-out test error as n_estimators increases, repeating randomized train/test experiments and averaging results. In the reported comparison, training error steadily decreased as ensembles grew, while test error decreased and then stabilized. All three tree ensembles strongly outperformed a single decision tree, illustrating how aggregation can turn unstable trees into more robust models. section 2.5 section 2.5.2
Figure 2.11 Training and test performance of bagging, random forest, and Extra Trees as ensemble size increases. Bagging used decision trees as the base estimator, random forest used randomized decision trees, and Extra Trees used extremely randomized trees.
Compare how bagging, random forests, and Extra Trees change training and test performance as the number of estimators grows. section 2.5.2
Extra Trees also extends to regression by using regression trees that make continuous-valued predictions. In scikit-learn, ExtraTreesRegressor can be trained with regression criteria such as squared error or absolute error; a bike-rental prediction workflow searches n_estimators and max_depth with cross-validation, then compares held-out metrics. For count-oriented prediction, the same regressor interface can use squared-error or Poisson criteria in the illustrated tuning workflow. section 7.2.1 section 7.4.3
For classification, scikit-learn provides ExtraTreesClassifier, including parallelization and out-of-bag estimation support. Its default bootstrap=False reflects the usual Extra Trees design. Do not confuse it with ExtraTreeClassifier: the singular class is an individual base-learning algorithm, whereas ExtraTreesClassifier is the ensemble. As with other ensembles, assess generalization on held-out data rather than relying only on training performance, and tune ensemble size and tree complexity for the task. section 2.4.3 section 2.5.2
