concept bagging in category machine learning

appears as: bagging, bagging
Ensemble Methods for Machine Learning

This is an excerpt from Manning's book Ensemble Methods for Machine Learning.

The class of homogeneous parallel ensemble methods includes two popular machine-learning methods, one or both of which you might have come across and even used before: bagging and random forest.

Recall that the two key components of an ensemble method are: ensemble diversity and model aggregation. Since homogeneous ensemble methods use the same learning algorithm on the same data set, how can they generate a set of diverse base estimators? They do this through random sampling of either the training examples (as bagging does) or features (as some variants of bagging do) or both (as random forest does).

Figure 2.2 Bagging, illustrated. Bagging uses bootstrap sampling to generate similar but not exactly identical subsets (observe the replicates above) from a single data set. Models are trained on each of these subsets resulting in similar but not exactly identical base estimators. During prediction, the individual base estimator predictions are aggregated into a final ensemble prediction. Also observe that training examples may repeat in the replicated subsets; this is a consequence of bootstrap sampling.

If we randomly sample both training examples and features (with or without replacement), we obtain a variant of bagging called Random Patches.

bag_ens = BaggingClassifier(
    base_estimator=SVC(), n_estimators=100, 
    max_samples=0.75, bootstrap=True,           #A
    max_features=0.5, bootstrap_features=True)  #B

Note that in the examples above, the base estimator is the support vector classifier, sklearn.svm.SVC. In general, random subspaces and random patches can be applied to any base learner to improve estimator diversity. As with bagging, in practice, increasing diversity tends to increase bias slightly.