Prediction averaging

Q: How deeply does Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI cover prediction averaging?

Explains averaging, voting, sampling variants, variance-bias trade-offs, and computational cost without deriving ensemble internals.

As explained in Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI

An averaging ensemble combines predictions from multiple models by averaging their outputs or taking a majority vote.

Prediction averaging combines multiple models to produce one prediction. For regression, it averages outputs; for classification, it uses majority voting. The models differ because they train on different samples, features, or both, making the combined estimate more reliable than a single model. section 5.1.1

Common averaging strategies differ in how they vary the training data:

Pasting trains models on subsamples drawn without replacement, then pools their predictions. section 5.1.1
Bagging uses bootstrapped samples, or sampling with replacement, to create diverse models. section 5.1.1
Random subspaces sample features, while random patches sample both training examples and features. section 5.1.1
These approaches primarily reduce prediction variance, although sampling can increase bias by excluding parts of the data distribution and can raise computational cost. section 5.6

Table 5.1 Bagging and sampling strategies (view table figure)

Averaging strategy	What happens to data	Parameters for BaggingClassifier/BaggingRegressor
Pasting	Training examples are sampled without replacement	`bootstrap = False` `max_samples < 1.0` `max_features = 1.0`
Bagging	Training examples are sampled with replacement (bootstrapping)	`bootstrap = True` `max_samples = 1.0` `max_features = 1.0`
Random subspaces	Features are sampled (without replacement)	`bootstrap = False` `max_samples = 1.0` `max_features < 1.0`
Random patches	Training examples and features are sampled without replacement	`bootstrap = False` `max_samples < 1.0` `max_features < 1.0`

Compare pasting, bagging, random subspaces, and random patches through their sampling choices and BaggingClassifier or BaggingRegressor parameters. section 5.1.1

A bagged tree classifier

A practical implementation uses scikit-learn's BaggingClassifier with a DecisionTreeClassifier, 300 estimators, bootstrap sampling, and five-fold shuffled cross-validation. The reported result is 0.809 accuracy, with fit and prediction timings included, making both predictive performance and operating cost visible. section 5.1.1

How Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI covers it

Cover of Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI

liveBook manning.com

Read Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Outlier Detection in PythonScore averaging is the aggregation of multiple detector scores or scores obtained under multiple parameter values by computing their mean.

Compare how 2 books explain prediction averaging →

Related concepts

baggingused by · Bagging, or bootstrap aggregation, is an ensemble technique that trains multiple models on bootstrap samples of data and combines their predictions.
varianceaddresses · Variance is the expected squared deviation of a random variable from its mean, measuring the spread of its distribution.
max aggregatoralternative to · A max aggregator combines multiple vectors by selecting the maximum value independently in each feature dimension.
model aggregationimplements · Model aggregation combines predictions from multiple predictive models into a single output, often by averaging them.
pastingincludes · Pasting is an ensemble technique that trains models on subsets sampled without replacement and combines their predictions.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI. The section links point to the parts of the book it draws on.