AdaBoost

As explained in Grokking Machine Learning

An adaptive boosting algorithm that combines sequentially trained weak learners by weighting their contributions according to predictive performance.

AdaBoost, short for adaptive boosting, combines weak learners into a stronger classifier. Unlike bagging, it trains learners sequentially: each new learner focuses on points misclassified by earlier ones. section 12.1 section 12.3

A classification example builds the model in these steps:

• Assign each point an initial weight of 1, then fit a depth-1 decision tree to the full dataset. section 12.3
• Increase the weights of misclassified points and train another learner on the reweighted dataset. section 12.3
• Score learners using log-odds based on accuracy; combine their predictions as weighted votes to produce the final classification. section 12.3

Classifying spam

A Scikit-Learn example trains six weak learners on email features counting the words “lottery” and “spam.” An exercise asks for a boosted prediction from three learners’ weights and votes. section 12.3 section 12.8

Figure 12.14 How to combine the weak learners into a strong learner in the AdaBoost model. We score each of the weak learners using the log-odds and make them vote based on their scores (the larger the score, the more voting power that particular learner has). Each of the regions in the bottom diagram has the sum of the scores of the weak learners. Note that to simplify our calculations, the predictions from the weak learners are +1 and –1, instead of 1 and 0.

Visualizes how log-odds scores give weak learners different voting power and combine their predictions. section 12.3

How Grokking Machine Learning covers it

How deep it goes

Explains sequential reweighting, depth-1 trees, log-odds voting, and a classification walkthrough; mentions regression without developing that case.

Examples

Uses triangle-and-square datasets, spam features, weighted-vote exercises, and a Scikit-Learn classifier example.

What you can do after

Apply it via Scikit-Learn to train an AdaBoost classifier and score its fit on labeled data.

How it connects

Contrasts boosting with bagging and connects AdaBoost to decision trees and ensemble learning.

liveBookmanning.com

Read Grokking Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningAdaBoost is an adaptive boosting technique that combines sequentially trained weak learners while increasing emphasis on previously misclassified examples.

Compare how 2 books explain AdaBoost →

Related concepts

• weak learneruses · A weak learner is a predictive model that performs only modestly better than chance or a simple baseline and can contribute to a stronger ensemble.
• exponential lossuses · Exponential loss is a classification loss that increases exponentially as an example’s prediction margin becomes more incorrect.
• reweightinguses · Reweighting is a technique that changes the relative influence of examples, groups, or outcomes by assigning them different weights during learning or analysis.
• sequential ensemble methodkind of · A sequential ensemble method trains base learners in successive stages, with later learners depending on or correcting the outputs of earlier learners.
• boostingkind of · Boosting is an ensemble-learning technique that builds predictive models sequentially and combines them, with later models improving on weaknesses of earlier ones.
• decision stumpuses · A decision stump is a decision tree of depth one with a single decision node and prediction leaves.

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning. The section links point to the parts of the book it draws on.