Boosting

As explained in Ensemble Methods for Machine Learning

Boosting is an ensemble technique that combines weak learners trained sequentially so that later learners improve on earlier mistakes.

Boosting combines simple, imperfect models into a stronger learner. Instead of training each model independently, it trains them sequentially so each new model improves on mistakes made earlier. The intuition resembles coordinating uncertain amateurs: individually limited, they become effective when their efforts are combined. section 4.0 section 4.1

A boosting ensemble follows this general pattern:

• Start with a weak learner: a simple classifier that performs only slightly better than random guessing. section 4.1
• Train the next learner with dependence on the previous learner, directing it toward earlier mistakes. section 1.5 section 4.1
• Repeat the process and combine the weak learners into one strong learner. section 4.1
• In adaptive boosting, or AdaBoost, reweight examples adaptively so successive learners focus on correcting previous errors. section 1.5 section 4.0

Decision stumps as weak learners

Decision trees are common boosting base estimators, especially decision stumps—trees of depth one. Increasing tree depth can improve the individual classifier, but excessive strength can cause overfitting and reduce deployed prediction performance. section 4.1 section 4.1

Figure 4.1 Differences between parallel and sequential ensembles: (1) base estimators in parallel ensembles are trained independently of each other, while in sequential ensembles, they are trained to improve on the predictions of the previous base estimator; (2) sequential ensembles typically use weak learners as base estimators.

Compare independent training in parallel ensembles with sequential training that improves on previous predictions. section 4.1

How Ensemble Methods for Machine Learning covers it

How deep it goes

Explains sequential dependence, weak learners, and adaptive reweighting, but stops short of full update equations.

Examples

Uses decision stumps, shallow decision trees, adaptive example weights, and the amateurs-versus-professionals analogy.

What it assumes

Assumes familiarity with ensemble methods and classifiers while defining weak learners through performance above random guessing.

What you can do after

Build from scratch a sequential ensemble that uses weak learners and adaptive reweighting to correct earlier mistakes.

How it connects

Contrasts boosting with bagging, random forests, and stacking, and connects it to AdaBoost and gradient boosting.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningAn ensemble technique that trains learners sequentially so later learners focus on weaknesses of earlier learners and combines their predictions.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthBoosting is an ensemble learning technique that combines weak learners trained sequentially with greater emphasis on previously misclassified examples.

Compare how 3 books explain boosting →

Related concepts

• weak learneruses · A weak learner is a predictive model that performs only modestly better than chance or a simple baseline and can contribute to a stronger ensemble.
• gradient boostingincludes · Gradient boosting is an ensemble-learning technique that sequentially adds predictive models, typically weak learners, to reduce a loss by fitting its gradient.
• AdaBoostincludes · AdaBoost is an ensemble-learning algorithm that sequentially combines weak learners, adaptively reweighting training examples so that later learners emphasize those earlier learners handled poorly.
• ensemblekind of · An ensemble is a technique that combines or considers outputs from multiple models or component methods collectively to produce or assess a joint result.
• baggingcontrasts with · Bagging, or bootstrap aggregation, is an ensemble technique that trains multiple models on bootstrap samples of data and combines their predictions.
• strong learnerproduces · A strong learner is a predictive model with substantially improved task performance, often formed by combining multiple weak learners.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.