AdaBoost

As explained in Ensemble Methods for Machine Learning

AdaBoost is an adaptive boosting technique that combines sequentially trained weak learners while increasing emphasis on previously misclassified examples.

AdaBoost, short for adaptive boosting, turns a sequence of weak classifiers into a stronger classifier. Rather than training estimators independently, it trains each new learner to focus on mistakes made earlier. A decision stump or shallow tree may be only slightly better than random guessing alone; together, such learners can form a useful nonlinear decision boundary. section 4.0 section 4.2

Figure 4.3 Initially (iteration 1), all the training examples are weighted equally (and hence plotted with the same size on the left). The decision stump learned on this data set is shown on the right. The correctly classified examples are plotted with smaller markers compared to the misclassified examples, which are plotted with larger markers. 

Start with equal importance for every training example. After a stump makes errors, the misclassified points receive more weight, so the next stump gives them priority. Repeating this handoff makes successive weak learners different and complementary. section 4.2.1

At each iteration, AdaBoost follows a focused training loop:

• Normalize the current example weights and fit a weak learner using those weighted examples. section 4.2.2
• Measure the learner’s weighted training error, then assign it an ensemble weight: lower-error learners contribute more. section 4.2.2
• Increase weights for examples the learner misclassifies and decrease weights for correctly classified examples. section 4.2.1
• Combine all learned classifiers by a weighted additive vote to make the final prediction. section 4.2.2

From three stumps to one classifier

On a two-class data set, the first stump can leave a small group wrong. AdaBoost enlarges that group’s influence for the second stump, then repeats the adjustment for the third. The resulting weighted ensemble can classify regions that no individual stump captures well. section 4.2.1

A small implementation needs a weighted base learner, such as a depth-one DecisionTreeClassifier trained with sample_weight. In scikit-learn, AdaBoostClassifier also supports multiclass classification and alternative weak learners. Tune n_estimators together with learning_rate: more estimators can keep example weights growing, while the learning rate shrinks each learner’s contribution. section 4.2.2 section 4.2.3

Adaptation is also AdaBoost’s main risk. Noisy outliers are often repeatedly misclassified, so their weights can dominate later training. The ensemble may either keep failing on an outlier or overfit it; normalization alone does not remove such points. Detecting and removing outliers during preprocessing can make training more robust. section 4.3

Handwritten digits

For 10-class digit recognition, use shallow trees as base estimators and select learning_rate and n_estimators jointly with five-fold cross-validation and grid search. Balanced accuracy evaluates parameter combinations across classes; a classification report and confusion matrix then expose which digits remain difficult. section 4.4 section 4.4.2

How Ensemble Methods for Machine Learning covers it

How deep it goes

Works through weighted fitting, error-based learner weighting, prediction, scikit-learn usage, tuning, and outlier limitations without formal proofs.

Examples

Uses weighted decision stumps, make_moons, breast cancer and iris classification, and handwritten-digit classification.

What it assumes

Assumes familiarity with classification, training error, decision trees, and train-test evaluation while reinforcing weak learners and sample weights.

What you can do after

Build it from scratch or apply it through AdaBoostClassifier, then tune estimator count and learning rate with cross-validation.

How it connects

Connects boosting to bagging, random forests, stacking, decision trees, cross-validation, grid search, exponential loss, and LogitBoost.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningAn adaptive boosting algorithm that combines sequentially trained weak learners by weighting their contributions according to predictive performance.

Compare how 2 books explain AdaBoost →

Related concepts

• weak learneruses · A weak learner is a predictive model that performs only modestly better than chance or a simple baseline and can contribute to a stronger ensemble.
• exponential lossuses · Exponential loss is a classification loss that increases exponentially as an example’s prediction margin becomes more incorrect.
• reweightinguses · Reweighting is a technique that changes the relative influence of examples, groups, or outcomes by assigning them different weights during learning or analysis.
• sequential ensemble methodkind of · A sequential ensemble method trains base learners in successive stages, with later learners depending on or correcting the outputs of earlier learners.
• boostingkind of · Boosting is an ensemble-learning technique that builds predictive models sequentially and combines them, with later models improving on weaknesses of earlier ones.
• decision stumpuses · A decision stump is a decision tree of depth one with a single decision node and prediction leaves.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.