ExtraTreesClassifier builds an ensemble of extremely randomized decision trees for classification. Each tree examines a random subset of features and chooses a random splitting threshold, creating diversity without relying on bootstrap sampling. section 2.4.3
The extra randomness can make training efficient, especially for high-dimensional data with many continuous features. In scikit-learn, bootstrap=False by default because random features and thresholds already diversify the trees. section 2.4.3
Configure the ensemble by balancing tree complexity and ensemble size:
- Increase n_estimators to use more trees; comparisons examine ensemble sizes from 2 through 19. section 2.5.2
- Control base-tree complexity with max_leaf_nodes, max_depth, impurity criteria, and related parameters. section 2.5.2
- Avoid confusing ExtraTreesClassifier with ExtraTreeClassifier: the plural class is the ensemble, while the singular class is an individual base learner. section 2.4.3
Complexity and ensemble-size experiments
Compare ExtraTreesClassifier with BaggingClassifier and RandomForestClassifier while varying max_leaf_nodes from 2 to 32 and holding n_estimators at 10. A separate experiment tracks training and test errors as the number of trees increases. section 2.5.2
Listing 2.5 Training and test errors with increasing ensemble size
max_leaf_nodes = 8 #1
n_runs = 20
n_estimator_range = range(2, 20, 1)
bag_trn_error = \
np.zeros((n_runs, len(n_estimator_range))) #2
rf_trn_error = \ #2
np.zeros((n_runs, len(n_estimator_range))) #2
xt_trn_error = \ #2
np.zeros((n_runs, len(n_estimator_range))) #2
bag_tst_error = \
np.zeros((n_runs, len(n_estimator_range))) #3
rf_tst_error = \ #3
np.zeros((n_runs, len(n_estimator_range))) #3
xt_tst_error = #3
np.zeros((n_runs, len(n_estimator_range))) #3
for run in range(0, n_runs):
X_trn, X_tst, y_trn, y_tst = train_test_split( #4
X, y, test_size=0.25,random_state=rng)
for j, n_estimators in enumerate(n_estimator_range):
tree = DecisionTreeClassifier( #5
max_leaf_nodes=max_leaf_nodes)
bag = BaggingClassifier(base_estimator=tree,
n_estimators=n_estimators,
max_samples=0.5, n_jobs=-1,
random_state=rng)
bag.fit(X_trn, y_trn)
bag_trn_error[run, j] = 1 - accuracy_score(y_trn, bag.predict(X_trn))
bag_tst_error[run, j] = 1 - accuracy_score(y_tst, bag.predict(X_tst))
rf = RandomForestClassifier( #6
max_leaf_nodes=max_leaf_nodes, n_estimators=n_estimators,
n_jobs=-1, random_state=rng)
rf.fit(X_trn, y_trn)
rf_trn_error[run, j] = 1 - accuracy_score(y_trn, rf.predict(X_trn))
rf_tst_error[run, j] = 1 - accuracy_score(y_tst, rf.predict(X_tst))
xt = ExtraTreesClassifier( #7
max_leaf_nodes=max_leaf_nodes, n_estimators=n_estimators,
bootstrap=True, n_jobs=-1, random_state=rng)
xt.fit(X_trn, y_trn)
xt_trn_error[run, j] = 1 - accuracy_score(y_trn, xt.predict(X_trn))
xt_tst_error[run, j] = 1 - accuracy_score(y_tst, xt.predict(X_tst))
Inspect the experiment structure for recording training and test errors across repeated runs and increasing ensemble sizes. section 2.5.2
