Gradient boosting
Gradient boosting is an ensemble-learning technique that sequentially adds predictive models fitted to reduce a loss by following its gradient.
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
A decision-tree ensemble model built by sequentially adding trees to optimize a predictive loss using gradient-based optimization.
- How it is used
- Applies sequential tree ensembles as a leading baseline for tabular classification, regression, ranking, and click-prediction tasks.
- How deep it goes
- Explains sequential corrections, logits, probabilities, tree controls, tuning tradeoffs, and inspectable XGBoost and LightGBM structures.
- Examples
- Demonstrates Airbnb NYC price classification, predicted-probability histograms, regression behavior, Scikit-learn cross-validation, and XGBoost and LightGBM tree extraction.
- What you can do after
- Apply it via Scikit-learn, XGBoost, or LightGBM to train, tune, evaluate, inspect, and compare tabular predictive models.
Ensemble Methods for Machine Learning
Gradient boosting is an ensemble-learning technique that sequentially adds models trained to correct the errors of prior models using gradient-based optimization.
- How it is used
- Builds sequential tree ensembles that correct residual errors for classification, regression, and ranking.
- How deep it goes
- Explains residual intuition, gradient-descent correspondence, stepwise training, from-scratch trees, and LightGBM tuning.
- Examples
- Uses noisy two-moons classification and AutoMPG regression with Huber-loss LightGBM.
- What you can do after
- Build it from scratch with weak regression trees, residual fitting, line search, and additive prediction updates.
Read how Ensemble Methods for Machine Learning explains gradient boosting →
Grokking Machine Learning
An ensemble model that sequentially combines decision trees, with each tree improving the ensemble's fit to the data.
- How it is used
- Combines sequential decision trees into a strong learner and serves as a candidate in a real-world model comparison.
- How deep it goes
- Works through residual fitting and learning-rate updates; leaves classification adjustments to supplementary resources.
- Examples
- Fits weekly app-engagement days from user age and demonstrates classifier training on Titanic data.
- What you can do after
- Apply it via Scikit-Learn’s gradient-boosting classifier, then compare model performance using accuracy or F1-score.
Read how Grokking Machine Learning explains gradient boosting →
Build AI Drug Discovery Pipelines
Gradient boosting is an ensemble learning technique that sequentially adds weak models trained to reduce the current ensemble's loss or errors.
- How it is used
- Applies sequential ensemble learning to QSAR classification, especially when molecular descriptor data are tabular.
- How deep it goes
- Explains weak learners, residual-based updates, and the gradient rationale, then demonstrates XGBClassifier without detailing hyperparameter tuning.
- Examples
- Uses RNA-binding versus protein-binding compounds, Mordred descriptors, XGBClassifier, AUPRC, and feature importance by gain, coverage, and frequency.
- What you can do after
- Apply it via XGBClassifier to classify molecular compounds with stratified cross-validation and precision-recall evaluation.
Read how Build AI Drug Discovery Pipelines explains gradient boosting →
Transfer Learning for Natural Language Processing
An ensemble method that iteratively adds decision-tree-based models to address weaknesses in earlier models.
- How it is used
- Provides a decision-tree-based classification baseline for email spam and movie review tasks.
- How deep it goes
- Explains iterative learning and demonstrates classifier training and evaluation without deriving the boosting algorithm.
- Examples
- Uses email spam and IMDB movie review classification alongside classifier reports and an email ROC curve.
- What you can do after
- Apply gradient boosting via scikit-learn and assess a classifier using cross-validation and test metrics.
Read how Transfer Learning for Natural Language Processing explains gradient boosting →
Fight Fraud with Machine Learning
Gradient boosted trees are an ensemble model that sequentially trains decision trees, with each tree correcting errors from earlier trees.
- How it is used
- Detects fraudulent online transactions as a stronger tree-based alternative to random forest.
- How deep it goes
- Explains sequential error correction, hyperparameter tuning, feature importance, tree plots, and score aggregation into probabilities.
- Examples
- Uses XGBoost listings, four-tree plots, feature-importance plots, and an amount-threshold trace for transaction classification.
- What you can do after
- Apply it via XGBoost to train, tune, compare, and interpret a fraud classifier.
Read how Fight Fraud with Machine Learning explains gradient boosting →
Machine Learning System Design
A gradient-boosting model is an ensemble model built by sequentially fitting weak predictive models to reduce the errors of the current ensemble.
- How it is used
- Uses gradient boosting to improve retail demand forecasts while reducing the gap between delivered and sold items.
- How deep it goes
- Explains convergence, tree-count selection, and feature complexity, but does not derive the boosting algorithm mathematically.
- Examples
- Evaluates six quantile models for retail sales using convergence curves, tree-count grids, lag counts, and window sizes.
- What you can do after
- Evaluate gradient-boosting experiments with convergence checks, baseline comparisons, and model-complexity curves.
Read how Machine Learning System Design explains gradient boosting →
The Art of AI Product Development
A gradient boosting algorithm builds an ensemble model by sequentially adding weak predictive models that reduce errors made by the existing ensemble.
- How it is used
- Supports model selection by balancing forecast accuracy, runtime, computational cost, and client-defined error tolerance.
- How deep it goes
- Compares practical performance and runtime, but stops short of explaining boosting mechanics or implementation details.
- Examples
- Uses demand forecasts, a ±5% error margin, and runtime-versus-performance comparisons across models.
- What you can do after
- Evaluate whether gradient boosting satisfies a defined error threshold while reducing training and execution time.
Read how The Art of AI Product Development explains gradient boosting →
Also covered in
- Machine Learning Algorithms in Depth
- Digital Twins in Action
- Grokking Machine Learning, Second EditionIn early access (MEAP)
What to know first
- random forestA random forest is an ensemble learning model that aggregates predictions from multiple decision trees trained with randomized data samples, feature subsets, or both.
- decision treeA decision tree is a tree-structured model or representation in which successive tests or choices lead through branches to outcomes, predictions, or actions.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- linear regressionLinear regression is a statistical method that models a numeric response as a linear function of one or more predictor variables with fitted parameters.
Related concepts
- XGBoostimplemented by · XGBoost is an open-source software library for efficient, scalable gradient-boosted decision-tree algorithms.
- Newton boostingcontrasts with · Newton boosting is a boosting approach that uses first- and second-order loss information to construct successive learners.
- decision treeuses · A decision tree is a tree-structured model or representation in which successive tests or choices lead through branches to outcomes, predictions, or actions.
- ensemblekind of · An ensemble is a technique or model that combines or considers outputs from multiple models, methods, or components to produce or assess a joint result.
- boostingkind of · Boosting is an ensemble-learning technique that builds predictive models sequentially and combines them, with later models improving on weaknesses of earlier ones.
Questions about gradient boosting
What is gradient boosting?
Gradient boosting is an ensemble-learning technique that sequentially adds predictive models fitted to reduce a loss by following its gradient.
Which Manning books explain gradient boosting?
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI, Ensemble Methods for Machine Learning, Grokking Machine Learning, Build AI Drug Discovery Pipelines, Transfer Learning for Natural Language Processing, Fight Fraud with Machine Learning, Machine Learning System Design and The Art of AI Product Development.
What should I know before learning gradient boosting?
Manning books that teach gradient boosting usually introduce random forest, decision tree, language model and linear regression first.
What is the difference between gradient boosting and Newton boosting?
Gradient boosting is an ensemble-learning technique that sequentially adds predictive models fitted to reduce a loss by following its gradient. Newton boosting is a boosting approach that uses first- and second-order loss information to construct successive learners.
What kinds of gradient boosting are there?
Kinds of gradient boosting include stochastic gradient boosting.
What is gradient boosting a kind of?
Gradient boosting is a kind of ensemble. An ensemble is a technique or model that combines or considers outputs from multiple models, methods, or components to produce or assess a joint result.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.







