liveBook · Manning

Hessian weighting

As explained in Ensemble Methods for Machine Learning

A Hessian-weighted gradient is a loss gradient scaled using second-order curvature information from the Hessian.

Hessian weighting scales a loss gradient by second-derivative curvature, giving difficult examples more influence when sequential ensembles fit their next weak learner. section 7.3 section 6.2.3

Apply the weighting inside each Newton boosting iteration:

  • Compute each example’s first derivative g and second derivative H of the loss. section 6.2.3
  • Form the residual r = −g/H, rather than using the negative gradient alone. section 6.2.3
  • Fit a shallow regression tree to the weighted residuals, find a step length by line search, and update the ensemble. section 6.2.3

Newton versus gradient boosting

On a noisy two-moons classification split with 25 estimators, the implementation reports approximately 8% test error for Newton boosting versus 12% for gradient boosting. section 6.2.3

initialize: F = f0, some constant value
for t = 1 to T:
1. compute first and second derivatives for each example,

The iteration outline shows where first- and second-derivative information enters the shared boosting procedure. section 6.2.3

How Ensemble Methods for Machine Learning covers it

How deep it goes
Explains the residual formula, iterative training procedure, visual effect, and comparison with gradient boosting.
Examples
Uses shallow regression trees, a noisy two-moons dataset, residual visualizations, and a 25-estimator error comparison.
What it assumes
Assumes familiarity with loss gradients, boosting, residuals, shallow regression trees, and basic second-derivative terminology.
What you can do after
Build it from scratch by implementing gradient and Newton boosting with selectable Hessian-weighted residuals.
How it connects
Connects loss functions, gradients, second derivatives, Newton boosting, gradient boosting, weak learners, and sequential ensembles.
Cover of Ensemble Methods for Machine Learning
liveBookmanning.com
Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.