Randomization deliberately injects variation into learning so an ensemble’s models do not all make the same predictions. In bagging, bootstrap samples create different training sets; in randomized trees, feature choices—and sometimes split thresholds—also vary. section 2.6 section 7.2.1
Common ways to apply randomization include:
- Bootstrap sampling draws training examples with replacement, then aggregates predictions from models trained on the resulting data replicates. section 2.6
- Random forests combine bootstrap samples with decision trees that evaluate only a random subset of features at each split. section 7.2.1
- Extra Trees randomize both the candidate feature and splitting threshold, allowing ensemble construction without bootstrap sampling. section 7.2.1
- Pasting samples without replacement, while random subspaces and random patches vary features alone or features plus training examples. section 2.6
Regression ensembles
On the AutoMPG data set, RandomForestRegressor and ExtraTreesRegressor can be trained with squared-error or absolute-error criteria, then compared using training and test MSE and MAE. section 7.2.1
Listing 7.3 Random forest and Extra Trees for regression
from sklearn.ensemble import RandomForestRegressor, ExtraTreesRegressor
from sklearn.metrics import mean_squared_error, mean_absolute_error
ensembles = { #1
'Random Forest MSE': RandomForestRegressor(criterion='squared_error'),
'Random Forest MAE': RandomForestRegressor(criterion='absolute_error'),
'ExtraTrees MSE': ExtraTreesRegressor(criterion='squared_error'),
'ExtraTrees MAE': ExtraTreesRegressor(criterion='absolute_error')}
results = pd.DataFrame() #2
ypred_trn = {}
ypred_tst = {}
for method, ensemble in ensembles.items():
ensemble.fit(Xtrn, ytrn) #3
ypred_trn[method] = ensemble.predict(Xtrn) #4
ypred_tst[method] = ensemble.predict(Xtst)
res = {'Method-Loss': method, #5
'Train MSE': mean_squared_error(ytrn, ypred_trn[method]),
'Train MAE': mean_absolute_error(ytrn, ypred_trn[method]),
'Test MSE': mean_squared_error(ytst, ypred_tst[method]),
'Test MAE': mean_absolute_error(ytst, ypred_tst[method])}
results = pd.concat([results,
pd.DataFrame.from_dict([res])], ignore_index=True)#6
The listing configures four scikit-learn regression ensembles, fits each model, generates training and test predictions, and records evaluation results. section 7.2.1
