Gradient boosting
Gradient boosting is an ensemble-learning technique that sequentially adds predictive models, typically weak learners, to reduce a loss by fitting its gradient.
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
A decision-tree ensemble model built by sequentially adding trees to optimize a predictive loss using gradient-based optimization.
- How it is used
- Applies sequential tree ensembles as a leading baseline for tabular classification, regression, ranking, and click-prediction tasks.
- How deep it goes
- Explains sequential corrections, logits, probabilities, tree controls, tuning tradeoffs, and inspectable XGBoost and LightGBM structures.
- Examples
- Demonstrates Airbnb NYC price classification, predicted-probability histograms, regression behavior, Scikit-learn cross-validation, and XGBoost and LightGBM tree extraction.
- What you can do after
- Apply it via Scikit-learn, XGBoost, or LightGBM to train, tune, evaluate, inspect, and compare tabular predictive models.
Ensemble Methods for Machine Learning
Gradient boosting is an ensemble-learning technique that sequentially adds models trained to correct the errors of prior models using gradient-based optimization.
- How it is used
- Builds sequential tree ensembles that correct residual errors for classification, regression, and ranking.
- How deep it goes
- Explains residual intuition, gradient-descent correspondence, stepwise training, from-scratch trees, and LightGBM tuning.
- Examples
- Uses noisy two-moons classification and AutoMPG regression with Huber-loss LightGBM.
- What you can do after
- Build it from scratch with weak regression trees, residual fitting, line search, and additive prediction updates.
Read how Ensemble Methods for Machine Learning explains gradient boosting →
Grokking Machine Learning
An ensemble model that sequentially combines decision trees, with each tree improving the ensemble's fit to the data.
- How it is used
- Combines sequential decision trees into a strong learner and serves as a candidate in a real-world model comparison.
- How deep it goes
- Works through residual fitting and learning-rate updates; leaves classification adjustments to supplementary resources.
- Examples
- Fits weekly app-engagement days from user age and demonstrates classifier training on Titanic data.
- What you can do after
- Apply it via Scikit-Learn’s gradient-boosting classifier, then compare model performance using accuracy or F1-score.
Read how Grokking Machine Learning explains gradient boosting →
Machine Learning System Design
A gradient-boosting model is an ensemble model built by sequentially fitting weak predictive models to reduce the errors of the current ensemble.
- How it is used
- Uses gradient boosting to improve retail demand forecasts while reducing the gap between delivered and sold items.
- How deep it goes
- Explains convergence, tree-count selection, and feature complexity, but does not derive the boosting algorithm mathematically.
- Examples
- Evaluates six quantile models for retail sales using convergence curves, tree-count grids, lag counts, and window sizes.
- What you can do after
- Evaluate gradient-boosting experiments with convergence checks, baseline comparisons, and model-complexity curves.
Read how Machine Learning System Design explains gradient boosting →
The Art of AI Product Development
A gradient boosting algorithm builds an ensemble model by sequentially adding weak predictive models that reduce errors made by the existing ensemble.
- How it is used
- Supports model selection by balancing forecast accuracy, runtime, computational cost, and client-defined error tolerance.
- How deep it goes
- Compares practical performance and runtime, but stops short of explaining boosting mechanics or implementation details.
- Examples
- Uses demand forecasts, a ±5% error margin, and runtime-versus-performance comparisons across models.
- What you can do after
- Evaluate whether gradient boosting satisfies a defined error threshold while reducing training and execution time.
Read how The Art of AI Product Development explains gradient boosting →
Related concepts
- XGBoostimplemented by · XGBoost is an open-source software library for efficient gradient-boosted decision-tree algorithms used in supervised machine learning.
- Newton boostingcontrasts with · Newton boosting is a boosting approach that uses first- and second-order loss information to construct successive learners.
- ensemblekind of · An ensemble is a technique that combines or considers outputs from multiple models or component methods collectively to produce or assess a joint result.
- random forestalternative to · A random forest is an ensemble model that combines predictions from multiple decision trees built with randomized data samples, feature selections, or both.
- decision treehas part · A decision tree is a branching structure that represents sequential tests or choices, with paths leading to outcomes, predictions, or actions.
- language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.