BaggingClassifier builds an ensemble by training base estimators on different bootstrap-sampled datasets, then aggregating their predictions. Because complex trees overfit different parts of the data, aggregation smooths individual mistakes and produces a more stable model. section 2.2.3
A typical configuration combines:
- A DecisionTreeClassifier base estimator, with complexity controlled through settings such as max_depth or max_leaf_nodes. section 2.2.3 section 2.5.2
- n_estimators to control ensemble size and max_samples to control the data used by each estimator. section 2.2.3
- oob_score=True to estimate generalization using samples excluded from each bootstrap training set. section 2.2.3
- n_jobs to parallelize independent training and prediction across CPUs; n_jobs=-1 uses all available CPUs. section 2.2.4
A scikit-learn configuration
Create a depth-limited decision tree, place 500 trees in the ensemble, sample 100 observations per estimator, enable OOB scoring, fit on training data, and predict on test data. An example reports an OOB score of 96.6% and test accuracy of 95.2%. section 2.2.3
Listing 2.3 Bagging with scikit-learn
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import BaggingClassifier
base_estimator = DecisionTreeClassifier(max_depth=10) #1
bag_ens = BaggingClassifier(base_estimator=base_estimator,
n_estimators=500, #2
max_samples=100, #3
oob_score=True, #4
random_state=rng)
bag_ens.fit(Xtrn, ytrn)
ypred = bag_ens.predict(Xtst)
Use the configuration listing to inspect how the base estimator, ensemble size, sample count, OOB scoring, and random state are connected. section 2.2.3
