Negative gradient

As explained in Ensemble Methods for Machine Learning

A negative gradient is the vector opposite to a function's gradient and indicates a local direction of decrease.

A negative gradient points in the direction that reduces a loss. In gradient boosting, it identifies how the current model should change rather than merely measuring whether predictions are wrong. section 5.2.1

Use the negative gradient to guide each sequential update:

• Compute the loss derivative for each training example. section 7.3
• Treat the negative gradients as residuals: large values mark examples requiring more attention, while small values indicate closer predictions. section 5.2.1
• Train a weak regressor to approximate those residuals, then add its prediction to the current model. section 5.2.1 section 7.3

Squared-loss example

With squared loss, the negative gradient is exactly the residual between the true label and the prediction. A large gap produces a large update signal; a small gap produces a small one. section 5.2.1

Figure 5.9 Comparing gradient descent (left) to gradient boosting (right). At iteration t, gradient descent updates the model using the negative gradient, -g_t. At iteration t, gradient boosting approximates the negative gradient by training a weak regressor, h_t, on the negative residuals -r^t_i. The step length α_t in gradient descent is equivalent to the hypothesis weight of each base estimator in a sequential ensemble.

Compare ordinary gradient descent, which uses the exact negative gradient, with gradient boosting, which approximates it using a weak regression tree. section 5.2.1

How Ensemble Methods for Machine Learning covers it

How deep it goes

Explains the residual equivalence for squared loss and its approximation by weak regressors, without developing general derivative calculations.

Examples

Uses squared loss, residual-augmented examples, regression trees, and comparisons with AdaBoost and gradient descent.

What it assumes

Assumes familiarity with model predictions and loss functions while reinforcing residuals through squared-loss examples.

What you can do after

Apply it via gradient boosting to interpret residuals as update targets for successive weak regressors.

How it connects

Connects loss functions, residuals, gradient descent, gradient boosting, weak learners, and Hessian-weighted gradients.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Related concepts

• gradient boostingused by · Gradient boosting is an ensemble-learning technique that sequentially adds predictive models, typically weak learners, to reduce a loss by fitting its gradient.
• residual errorequivalent to · Residual error is the difference between an observed value and its corresponding value predicted or fitted by a model.
• gradient descentused by · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
• gradientcontrasts with · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.