Bootstrap sampling creates multiple datasets by drawing observations with replacement from an observed sample. The resamples approximate the underlying population without requiring an explicit distribution, making the technique useful for estimating statistical uncertainty and model-performance variability. section 5.1.1
In machine learning, bootstrap samples also create diverse training sets for ensemble models. Less-correlated models generally make averaging more effective, while aggressive subsampling can introduce bias or omit important parts of the data. section 5.1.1 section 5.6
Use bootstrap sampling to:
- Estimate the uncertainty or sampling distribution of a statistic. section 5.1.1
- Train multiple models on varied datasets and pool their predictions through bagging. section 5.1.1 section 5.6
- Distinguish bagging from pasting: bagging samples with replacement, whereas pasting samples without replacement. section 5.1.1
Bagged tree classifier
A scikit-learn BaggingClassifier can use decision trees with bootstrap=True, max_samples=1.0, max_features=1.0, and multiple estimators, creating an ensemble from bootstrapped training data.
