Hessian matrix

As explained in Ensemble Methods for Machine Learning

A Hessian is the square matrix of second-order partial derivatives of a scalar-valued function.

A Hessian matrix captures how the slope of a scalar-valued function changes in every pair of input directions. Where a gradient gives one first derivative per variable, the Hessian differentiates every gradient entry again. For a function of x, y, and z, this produces a square matrix of second-order partial derivatives; its diagonal entries describe repeated differentiation with respect to one variable, while off-diagonal entries describe cross-variable change. The matrix is symmetric when reversing differentiation order gives the same result. section 6.1

Its practical value is curvature information. Gradient descent uses a local linear approximation and moves according to the gradient alone. Newton’s method combines the gradient with the inverse Hessian, forming a local quadratic approximation that can provide a more informative descent direction. In the multivariate update, the inverse Hessian plays the role that division by the second derivative plays for a one-variable function. section 6.1 section 6.1.1

To use a Hessian in Newton-style minimization:

• Evaluate the gradient vector and Hessian matrix at the current estimate. section 6.1.1
• Multiply the gradient by the inverse Hessian and negate the result to obtain the descent direction. section 6.1.1
• Use a step length of 1 for Newton’s method, update the estimate, and stop when successive estimates are sufficiently close. section 6.1.1

Following the Branin surface

For the two-variable Branin function, the Hessian is computed by analytically differentiating the gradient vector. Starting at [2, -5], Newton’s descent uses that Hessian and the gradient at each iteration to move toward a local minimum. Compared with gradient descent from the same start, its path is more direct because each update uses a second-order local approximation. A zero gradient can also occur at a saddle point, so descent methods can converge there as well. section 6.1.1

Hessians also guide sequential ensemble learning. A custom LightGBM objective can return both a loss gradient and Hessian; the Hessian is used while learning tree leaf nodes. With focal loss, first and second derivatives may be approximated numerically, provided the loss, metric, and objective accept array-like labels and predictions. In Newton boosting, Hessian-weighted gradients serve as residuals and use curvature to increase emphasis on higher-loss examples. section 5.4.3 section 7.3

Figure 6.3 We compare the solution paths of Newton’s descent and gradient descent starting from [2,-5] (square) and both converging to one of the local minima (circle). Newton’s descent (solid line) progresses toward the local minimum in a more direct way compared to gradient descent (dotted line). This is because Newton’s descent uses a more informative second-order local approximation with each update, while gradient descent only uses a first-order local approximation.

Compare Newton and gradient-descent paths from the same starting point to see how second-order curvature information changes the update direction. section 6.1.1

How Ensemble Methods for Machine Learning covers it

How deep it goes

Explains multivariate construction, inverse-Hessian updates, analytic and numerical derivatives, and leaf-node use without detailing large-problem handling.

Examples

Uses a three-variable function, the Branin function, focal-loss derivatives, and LightGBM training on breast-cancer data.

What it assumes

Builds from first and second derivatives, then extends them from one-variable to multivariate functions.

What you can do after

Apply it via LightGBM custom objectives or implement Newton updates from gradients and Hessians.

How it connects

Connects gradients, Newton’s method, gradient descent, focal loss, LightGBM, and Newton boosting.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIThe square matrix of second-order partial derivatives of a scalar multivariable function, describing its local curvature.

Compare how 2 books explain Hessian matrix →

Related concepts

• GPTQused by · GPTQ is a one-shot post-training weight-quantization method that uses calibration data and approximate second-order information to reduce precision while limiting accuracy loss.
• newton-raphson methodused by · The Newton-Raphson method is an iterative numerical technique that uses first- and second-derivative information to find a root or optimize an objective.
• XGBoostused by · XGBoost is an open-source software library for efficient gradient-boosted decision-tree algorithms used in supervised machine learning.
• Jacobianequivalent to · The matrix of all first-order partial derivatives of a vector-valued function with respect to its inputs.
• Taylor seriesused by · A Taylor series is a power series whose coefficients are determined by a function’s derivatives evaluated at a specified expansion point.
• Newton boostinguses · Newton boosting is a boosting approach that uses first- and second-order loss information to construct successive learners.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.