Boosting

As explained in Grokking Machine Learning

An ensemble technique that trains learners sequentially so later learners focus on weaknesses of earlier learners and combines their predictions.

Could several modest models work together to make stronger predictions? Boosting trains weak learners in sequence: each new learner focuses more on examples earlier learners handled poorly, and their predictions combine into a stronger model. section 12.0 section 12.7

A basic boosting process works like this:

• Train an initial learner on the dataset. section 12.1
• Give more attention to examples the learner predicts poorly, then train another learner on the modified data. section 12.1
• Repeat the process and combine learner predictions; boosting can use a weighted vote. section 12.7

Bagging also combines weak learners, but trains them on random data subsets. Boosting instead builds learners strategically, one after another, to address earlier weaknesses. section 12.1

AdaBoost on spam email

AdaBoost is a boosting algorithm for classification and regression. In a spam-classification example, six depth-one decision trees use the number of occurrences of “lottery” and “spam” as features; Scikit-Learn fits the classifier to the labels. section 12.3

Figure 12.1 Ensemble methods consist of joining several weak learners to build a strong learner.

See how weak learners combine to form a strong learner. section 12.1

How Grokking Machine Learning covers it

How deep it goes

Explains AdaBoost’s reweighting and weighted voting with a worked classification example, without detailing gradient boosting or XGBoost.

Examples

Uses a triangle-and-square classification dataset and a spam-email example with “lottery” and “spam” features.

What you can do after

Apply it via Scikit-Learn’s AdaBoostClassifier to fit and score a spam classifier.

How it connects

Contrasts boosting with bagging and connects it to ensemble learning, AdaBoost, gradient boosting, XGBoost, and decision trees.

liveBookmanning.com

Read Grokking Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningBoosting is an ensemble technique that combines weak learners trained sequentially so that later learners improve on earlier mistakes.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthBoosting is an ensemble learning technique that combines weak learners trained sequentially with greater emphasis on previously misclassified examples.

Compare how 3 books explain boosting →

Related concepts

• weak learneruses · A weak learner is a predictive model that performs only modestly better than chance or a simple baseline and can contribute to a stronger ensemble.
• gradient boostingincludes · Gradient boosting is an ensemble-learning technique that sequentially adds predictive models, typically weak learners, to reduce a loss by fitting its gradient.
• AdaBoostincludes · AdaBoost is an ensemble-learning algorithm that sequentially combines weak learners, adaptively reweighting training examples so that later learners emphasize those earlier learners handled poorly.
• ensemblekind of · An ensemble is a technique that combines or considers outputs from multiple models or component methods collectively to produce or assess a joint result.
• baggingcontrasts with · Bagging, or bootstrap aggregation, is an ensemble technique that trains multiple models on bootstrap samples of data and combines their predictions.
• strong learnerproduces · A strong learner is a predictive model with substantially improved task performance, often formed by combining multiple weak learners.

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning. The section links point to the parts of the book it draws on.