liveBook · Manning

Boosting

As explained in Timeless Algorithms, The Foundational Ideas

Boosting is an ensemble technique that builds models sequentially, with later models focusing on cases earlier models handled poorly.

Boosting combines predictions from multiple limited models, trained sequentially. Each new model concentrates on observations earlier models handled poorly, correcting the ensemble step by step. section 8.8.3 section 9.2.3

A boosted classifier’s margin measures the combined vote’s strength for the correct class relative to the competing class. AdaBoost can continue increasing margins after training accuracy is perfect, offering one explanation for continued improvement on new data. This margin is not the geometric distance used by a support vector machine. section 8.8.3

Boosting and bagging build ensembles differently:

  • Bagging trains models independently on bootstrap samples; boosting trains sequentially through successive correction. section 9.2.3
  • AdaBoost, introduced in 1995, and later methods such as gradient boosting and XGBoost belong to the boosting family. section 9.2.3

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines sequential focus on earlier errors and interprets classifier margins, but does not detail training steps or equations.
Examples
Names AdaBoost, gradient boosting, and XGBoost as boosting methods, dating AdaBoost to 1995.
What you can do after
Understand how sequential correction differs from bagging and how boosting margins can grow after training accuracy is perfect.
How it connects
Connects boosting to ensemble learning, bagging, random forests, AdaBoost, and support vector machine margin reasoning.

Also explained in

Compare how 5 books explain boosting →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.