Newton boosting

As explained in Ensemble Methods for Machine Learning

Newton boosting is a boosting approach that uses first- and second-order loss information to construct successive learners.

Newton boosting builds a sequential ensemble by repeatedly directing a weak learner toward the examples the current model handles poorly. Its distinguishing idea is to use both the loss gradient and its Hessian: the gradient measures the direction of error, while the second derivative scales that correction. This produces weighted residuals, letting each new learner approximate a Newton step rather than an ordinary gradient step. section 6.0 section 6.2.1

Like gradient boosting, it can work with loss functions for classification, regression, and ranking. The ensemble still grows stage by stage: fit a weak regression tree to the current correction, add its prediction to the model, and repeat. The practical difference is how the correction is formed: gradient boosting uses negative gradients directly, whereas Newton boosting uses negative gradients scaled by Hessian information. section 6.2.1 section 6.6

A basic Newton-boosting iteration is:

• Compute the first and second derivatives of the loss for every training example. section 6.2.3
• Form a weighted residual, using the negative gradient divided by the Hessian. section 6.2.3
• Fit a weak decision-tree regressor to the features and weighted residuals. section 6.2.3
• Choose a step length with line search and add the scaled tree prediction to the current ensemble. section 6.2.3

Figure 6.8 Unweighted residuals (left) used by gradient boosting compared to Hessian-weighted residuals (right) used by Newton boosting. Positive values of misclassification margin (y ⋅ f(x) > 0) indicate correct classification. For misclassifications, we have y ⋅ f(x) < 0. For badly misclassified examples, the Hessian-weighted gradients capture that notion more effectively than unweighted gradients.

Compare ordinary residuals with Hessian-weighted residuals to see why the latter emphasize badly misclassified examples differently. section 6.2.1

A small classification comparison

On a noisy two-moons classification data set, an ensemble of 25 weak trees produced about 8% test error with Newton boosting versus about 12% with gradient boosting. The implementations share the same sequential structure; replacing negative-gradient residuals with Hessian-weighted residuals is the essential change. section 6.2.3

XGBoost applies Newton boosting to regularized tree ensembles. Its objective controls fit while also penalizing model complexity, including tree structure through the number of leaf nodes. Regularization, learning rate, early stopping, and tree depth help limit overfitting, but introduce parameters that commonly require cross-validation or randomized search. section 6.3 section 6.3.1 section 6.6

Second derivatives also constrain loss selection. For example, XGBoost cannot directly use MAE or Huber loss in this setting because their required second derivatives are unavailable; pseudo-Huber supplies a differentiable alternative. For large-scale work, XGBoost adds efficient split finding plus parallel and distributed processing. section 7.3.2 section 6.3

Regression with XGBoost

For continuous or count targets, choose a loss that matches the problem, tune tree, learning, and regularization parameters, and evaluate on held-out data. In a regression comparison, XGBoost models used squared-error, pseudo-Huber, Poisson, and Tweedie objectives; results varied by objective, reinforcing that loss selection matters. section 7.3 section 7.4.4

How Ensemble Methods for Machine Learning covers it

How deep it goes

Derives weighted residual intuition, specifies the training loop, and implements logistic-loss tree ensembles without detailing distributed internals.

Examples

Uses noisy two-moons classification, document retrieval, continuous regression, and bike-rental prediction with XGBoost.

What it assumes

Assumes familiarity with boosting, gradients, residuals, weak learners, and loss functions while introducing Hessian-weighted residuals.

What you can do after

Apply it via XGBoost by selecting differentiable objectives and tuning regularization, learning rate, tree depth, and early stopping.

How it connects

Connects Newton’s descent with gradient boosting, AdaBoost, regression trees, regularized loss functions, XGBoost, and early stopping.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Related concepts

• weighted residualused by · A weighted residual is a per-example prediction discrepancy scaled by a weight that represents the example's contribution to an optimization update.
• gradient boostingcontrasts with · Gradient boosting is an ensemble-learning technique that sequentially adds predictive models, typically weak learners, to reduce a loss by fitting its gradient.
• gblinear boosterimplemented by · The gblinear booster is an XGBoost configuration that uses regularized linear models as base learners.
• XGBoostimplemented by · XGBoost is an open-source software library for efficient gradient-boosted decision-tree algorithms used in supervised machine learning.
• newton-raphson methoduses · The Newton-Raphson method is an iterative numerical technique that uses first- and second-derivative information to find a root or optimize an objective.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.