Base-learning algorithm

As explained in Ensemble Methods for Machine Learning

A base-learning algorithm is an algorithm used to train an individual base estimator within an ensemble.

A base-learning algorithm trains one individual base estimator inside an ensemble. Each estimator forms an independent opinion, and the ensemble combines those predictions into a final result. section 1.4 section 1.5

Think of a diagnostic team: each resident applies a method to the same patient data, then a senior doctor aggregates their opinions. Model diversity comes from varying the algorithms or training data. section 2.1 section 7.2

A base-learning algorithm participates in an ensemble through a simple workflow:

• Choose one or more learning algorithms, such as decision-tree regression, support vector regression, kernel ridge regression, k-nearest neighbors, Gaussian processes, or neural networks. section 1.4
• Train each base estimator independently on the available training data. section 3.0 section 7.2
• Generate individual predictions, then combine them with averaging, voting, weighting, or a learned meta-estimator. section 1.4 section 3.5

Homogeneous ensembles keep the base-learning algorithm fixed and vary the training data, as in bagging and random forests. Heterogeneous ensembles change the algorithm itself, using combinations such as a decision tree, SVM, and neural network. section 3.0 section 1.5

A six-model regression ensemble

For Friedman-1 regression data, train kernel ridge, support vector, decision-tree, k-nearest-neighbor, Gaussian-process, and multilayer-perceptron regressors. Predict with all six models and average their outputs. Increasing the ensemble size can improve generalization while reducing performance variation across combinations. section 1.4

Figure 1.7 Our first ensemble method ensembles the predictions of six different regression models by averaging them. This simple ensemble illustrates two key principles of ensembling: (1) model diversity, achieved in this case by using six different base machine-learning models; and (2) model aggregation, achieved in this case by simple averaging across predictions.

Visualizes how different base-learning algorithms create diverse estimators whose predictions are combined by averaging. section 1.4

How Ensemble Methods for Machine Learning covers it

How deep it goes

Defines the role and contrasts diversity mechanisms, then demonstrates training and averaging six regressors without deriving algorithm internals.

Examples

Uses Friedman-1 regression with six algorithms and illustrates aggregation through a diagnostic-team analogy.

What it assumes

Assumes familiarity with individual machine-learning models, training data, predictions, and ensemble methods.

What you can do after

Operate/configure scikit-learn estimators, train diverse base models, and aggregate their predictions into an ensemble output.

How it connects

Connects base-learning algorithms to base estimators, model aggregation, homogeneous ensembles, heterogeneous ensembles, stacking, bagging, and random forests.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Related concepts

• heterogeneous ensembleused by · A heterogeneous ensemble is an ensemble whose component learners use different model types or learning algorithms.
• base learnerproduces · A base learner is an individual predictive model whose output contributes to an ensemble or higher-level estimator.
• learning algorithmkind of · A learning algorithm is a computational procedure that fits or improves a model by adjusting its parameters or otherwise optimizing its objective using data or experience.
• LogitBoostused by · LogitBoost is a boosting algorithm that constructs an ensemble by optimizing logistic loss, typically through probability-based classification updates.
• machine learningkind of · Machine learning is a field of computational methods in which systems learn patterns or relationships from data to make predictions, decisions, or other outputs.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.