Sequential ensemble method

As explained in Ensemble Methods for Machine Learning

A sequential ensemble is an ensemble whose base models are trained dependently in successive stages to reduce previous mistakes.

A sequential ensemble turns several modest models into a stronger predictor by training them in stages. Each new model depends on the previous model and concentrates on reducing its mistakes, unlike parallel ensembles whose models are trained independently. section 1.5 section 4.1

Think of weak learners as amateurs who are each slightly better than guessing. Sequential training lets the next amateur focus on the cases earlier learners handled poorly, gradually combining their limited strengths into one strong learner. section 4.1

A typical sequential ensemble proceeds stage by stage:

• Start with a simple base estimator, often a decision stump or shallow tree. section 4.1
• Train the current estimator using information about errors from the preceding estimator. section 1.5 section 7.3
• Repeat the process so later estimators improve the ensemble’s previous predictions. section 7.3
• Combine the learners into a stronger final predictor rather than relying on one complex model. section 4.1

Classification and regression

AdaBoost adapts later learners to classification mistakes, while gradient and Newton boosting train shallow regression trees against residuals. Residual magnitude identifies large errors; its sign indicates whether predictions should increase or decrease. section 4.2 section 7.3

Figure 4.1 Differences between parallel and sequential ensembles: (1) base estimators in parallel ensembles are trained independently of each other, while in sequential ensembles, they are trained to improve on the predictions of the previous base estimator; (2) sequential ensembles typically use weak learners as base estimators.

Compare independent parallel training with dependent sequential training, including the typical use of strong versus weak learners. section 4.1

How Ensemble Methods for Machine Learning covers it

How deep it goes

Explains dependent training, weak learners, error signals, and regression residuals without deriving a complete boosting algorithm.

Examples

Uses decision stumps, shallow regression trees, classification mistakes, and regression residuals as teaching examples.

What it assumes

Assumes familiarity with base estimators and prediction errors while defining weak learners and contrasting parallel training.

What you can do after

Apply it via AdaBoost, gradient boosting, Newton boosting, LightGBM, or XGBoost for classification and regression settings.

How it connects

Connects boosting with AdaBoost, gradient boosting, Newton boosting, weak learners, residuals, and parallel ensembles.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Related concepts

• AdaBoostincludes · AdaBoost is an ensemble-learning algorithm that sequentially combines weak learners, adaptively reweighting training examples so that later learners emphasize those earlier learners handled poorly.
• parallel ensemble methodcontrasts with · A parallel ensemble method combines outputs from multiple independently trained base learners that perform the same task.
• ensemblekind of · An ensemble is a technique that combines or considers outputs from multiple models or component methods collectively to produce or assess a joint result.
• strong learnerproduces · A strong learner is a predictive model with substantially improved task performance, often formed by combining multiple weak learners.
• weak learneruses · A weak learner is a predictive model that performs only modestly better than chance or a simple baseline and can contribute to a stronger ensemble.
• boostingincludes · Boosting is an ensemble-learning technique that builds predictive models sequentially and combines them, with later models improving on weaknesses of earlier ones.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.