Gradient boosting

As explained in Grokking Machine Learning

An ensemble model that sequentially combines decision trees, with each tree improving the ensemble's fit to the data.

Gradient boosting combines decision trees in sequence: each new tree focuses on what the existing model still predicts poorly. Together, the trees form a stronger learner. section 12.1 section 12.4

For regression, the fitting process works like this:

• Start with a simple prediction: under mean squared error, the first tree predicts the average label. section 12.4
• Calculate residuals—the differences between labels and current predictions—and fit the next tree to them. section 12.4
• Add the new tree’s predictions, scaled by the learning rate, to the ensemble. Repeat using the updated residuals. section 12.4

Predicting app engagement

Using users’ ages to predict days of weekly app use, the first tree predicts the average label, 4. Later trees fit residuals; with a learning rate of 0.8, their scaled predictions are added to the ensemble. section 12.4

Figure 12.19 The plot of the user engagement dataset from table 12.3. The horizontal axis represents the age of the users, and the vertical axis represents the days per week that the user uses our app.

Plots user age against weekly app-engagement days, the example dataset fitted by gradient boosting. section 12.4

Trees can have depth greater than one, unlike the depth-one weak learners used in the AdaBoost comparison. Gradient boosting also supports classification, with adjustments to the regression procedure. XGBoost is a related implementation that uses similarity scores to build trees and prunes splits that fail its threshold. section 12.4 section 12.5

Scikit-Learn provides gradient-boosting regressor and classifier APIs. A classifier can be trained on Titanic data and compared with other models using measures such as accuracy and F1-score. section 13.4

How Grokking Machine Learning covers it

How deep it goes

Works through residual fitting and learning-rate updates; leaves classification adjustments to supplementary resources.

Examples

Fits weekly app-engagement days from user age and demonstrates classifier training on Titanic data.

What you can do after

Apply it via Scikit-Learn’s gradient-boosting classifier, then compare model performance using accuracy or F1-score.

How it connects

Connects decision trees, residuals, mean squared error, AdaBoost, and XGBoost.

liveBookmanning.com

Read Grokking Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA decision-tree ensemble model built by sequentially adding trees to optimize a predictive loss using gradient-based optimization.
• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningGradient boosting is an ensemble-learning technique that sequentially adds models trained to correct the errors of prior models using gradient-based optimization.
• 

  liveBookmanning.com
  Machine Learning System DesignA gradient-boosting model is an ensemble model built by sequentially fitting weak predictive models to reduce the errors of the current ensemble.
• 

  liveBookmanning.com
  The Art of AI Product DevelopmentA gradient boosting algorithm builds an ensemble model by sequentially adding weak predictive models that reduce errors made by the existing ensemble.

Compare how 5 books explain gradient boosting →

Related concepts

• XGBoostimplemented by · XGBoost is an open-source software library for efficient gradient-boosted decision-tree algorithms used in supervised machine learning.
• Newton boostingcontrasts with · Newton boosting is a boosting approach that uses first- and second-order loss information to construct successive learners.
• ensemblekind of · An ensemble is a technique that combines or considers outputs from multiple models or component methods collectively to produce or assess a joint result.
• random forestalternative to · A random forest is an ensemble model that combines predictions from multiple decision trees built with randomized data samples, feature selections, or both.
• decision treehas part · A decision tree is a branching structure that represents sequential tests or choices, with paths leading to outcomes, predictions, or actions.
• language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning. The section links point to the parts of the book it draws on.