liveBook · Manning

Weighted residual

As explained in Ensemble Methods for Machine Learning

A weighted residual is a per-example prediction discrepancy scaled by a weight that represents the example's contribution to an optimization update.

A weighted residual combines two signals about each training example: the residual measures prediction error, while the weight measures how strongly that error should influence the next optimization update. Newton boosting uses this combination to focus new weak learners on examples needing the most attention. section 6.0 section 6.2.1

The residual is computed from the loss gradient, as in gradient boosting. The weight comes from the loss Hessian, or second derivative. Thus, a weighted residual is a Hessian-scaled gradient residual rather than an unweighted measure of error. section 6.2.1

Newton boosting uses weighted residuals procedurally:

  • Evaluate the current model on each training example. section 6.2.1
  • Compute each example’s loss gradient as its residual and its Hessian as its weight. section 6.2.1
  • Train a weak regression-tree estimator over these weighted residuals to approximate the Newton step. section 6.6
  • Update the ensemble, repeating the process so later learners correct earlier errors. section 6.2.1

Unweighted versus Hessian-weighted attention

For classification, the misclassification margin y · f(x) indicates whether an example is correctly classified. Gradient boosting uses unweighted residuals, whereas Newton boosting scales the gradients with Hessian information so badly misclassified examples are represented more effectively.

How Ensemble Methods for Machine Learning covers it

How deep it goes
Explains gradients, Hessians, weak-estimator fitting, and iteration mechanics, while deferring detailed derivations and computation.
Examples
Illustrates logistic-loss gradients, Hessians, classification margins, and comparisons between unweighted and Hessian-weighted residuals.
What it assumes
Assumes familiarity with boosting and gradients while introducing Hessian weighting through Newton's descent.
What you can do after
Build it from scratch with weak regression trees or apply it via XGBoost to classification, regression, and ranking.
How it connects
Connects AdaBoost weights, gradient boosting residuals, Newton's descent, regression trees, regularization, and XGBoost.
Cover of Ensemble Methods for Machine Learning
liveBookmanning.com
Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.