XGBoost

As explained in Grokking Machine Learning

An open-source software library that implements optimized gradient-boosted decision trees for machine-learning tasks.

XGBoost is an optimized implementation of gradient boosting that combines decision trees into a stronger learner. Each new tree focuses on the residual errors left by earlier trees, helping the ensemble improve predictions for regression and classification. section 12.1 section 12.5

A typical XGBoost training cycle works like this:

• Start with an initial prediction, such as 0.5 for every example. section 12.5
• Fit the next decision tree to the current residuals rather than directly repeating the original labels. section 12.5
• Choose splits using a similarity score, then prune branches whose similarity gain does not exceed the minimum split-loss threshold. section 12.5
• Add the new tree’s prediction after scaling it by the learning rate, and repeat with the smaller residuals. section 12.5

A small engagement model

For users aged 10–80, the first learner predicts 0.5 for everyone. Later trees refine those predictions from residuals such as 6.5, 4.5, and 0.5. A Python model can use XGBRegressor with three estimators, depth two, learning rate 0.7, and minimum split loss 1. section 12.5

Figure 12.1 Ensemble methods consist of joining several weak learners to build a strong learner.

Visualize how several weak learners join together to form a stronger ensemble. section 12.1

How Grokking Machine Learning covers it

How deep it goes

Explains residual fitting, similarity scores, split selection, pruning, learning rates, predictions, and a Python configuration.

Examples

Uses app engagement by age, residual tables, tree diagrams, pruning comparisons, prediction plots, and an XGBRegressor listing.

What it assumes

Assumes familiarity with decision trees, regression or classification, and the basic idea of combining weak learners.

What you can do after

Apply XGBoost via the xgboost package and inspect how successive trees reduce residual errors.

How it connects

Connects ensemble learning with gradient boosting, decision trees, residuals, similarity scores, pruning, learning rates, regression, and classification.

liveBookmanning.com

Read Grokking Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA predictive model trained using XGBoost's gradient-boosted decision-tree algorithms.
• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningXGBoost is an open-source machine-learning library implementing scalable gradient-boosted decision-tree algorithms.
• 

  liveBookmanning.com
  Machine Learning Platform EngineeringAn XGBoost model is a gradient-boosted decision-tree model trained using the XGBoost algorithm.
• 

  liveBookmanning.com
  Delivering Machine Learning Projects: From design to deploymentXGBoost is an open-source implementation of gradient-boosted decision trees for efficient supervised learning.
• 

  liveBookmanning.com
  Financial AI in PracticeXGBoost is an open-source software library for gradient-boosted decision trees.

Compare how 6 books explain XGBoost →

Related concepts

• gradient boostingimplements · Gradient boosting is an ensemble-learning technique that sequentially adds predictive models, typically weak learners, to reduce a loss by fitting its gradient.
• decision treeuses · A decision tree is a branching structure that represents sequential tests or choices, with paths leading to outcomes, predictions, or actions.
• early stoppinguses · Early stopping is a technique that terminates an iterative process before its planned completion when a specified condition indicates that further work is unnecessary or unlikely to improve the objective.
• LightGBMcontrasts with · LightGBM is an open-source gradient-boosting framework for efficiently training tree-based machine-learning models.
• weighted quantile sketchuses · A weighted quantile sketch is an approximate algorithm for estimating quantiles or quantile-based split points while accounting for item weights.

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning. The section links point to the parts of the book it draws on.