liveBook · Manning

Hessian matrix

A Hessian matrix is the square matrix of second-order partial derivatives of a scalar-valued function of multiple variables.

How 3 Manning books explain it

Ensemble Methods for Machine Learning

A Hessian is the square matrix of second-order partial derivatives of a scalar-valued function.

How it is used
Guides Newton’s method and Newton boosting with second-order curvature information.
How deep it goes
Explains multivariate construction, inverse-Hessian updates, analytic and numerical derivatives, and leaf-node use without detailing large-problem handling.
Examples
Uses a three-variable function, the Branin function, focal-loss derivatives, and LightGBM training on breast-cancer data.
What you can do after
Apply it via LightGBM custom objectives or implement Newton updates from gradients and Hessians.

Read how Ensemble Methods for Machine Learning explains Hessian matrix →

Quantization and Fast Inference

A Hessian is a matrix of second-order partial derivatives describing the local curvature of a scalar-valued function.

How it is used
Uses activation-dependent sensitivity to guide weight rounding during GPTQ quantization.
How deep it goes
Defines the Hessian through layer reconstruction loss, gives H = XXᵀ, and connects its inverse to error compensation.
Examples
Uses OPT-6.7B sensitivity outliers and compares GPTQ with round-to-nearest quantization at 3-bit and 4-bit.
What you can do after
Apply it via GPTQ to guide weight quantization using calibration activations and compensate rounding errors.

Read how Quantization and Fast Inference explains Hessian matrix →

Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI

The square matrix of second-order partial derivatives of a scalar multivariable function, describing its local curvature.

How it is used
Uses second-order curvature information to adapt gradient-boosting updates and accelerate Newton-style optimization.
How deep it goes
Explains curvature, update behavior, regularization, and implementation, but defers detailed analytical and numerical derivations.
Examples
Demonstrates Hessian-weighted updates in a custom NewtonianGradientBoosting class and reports 0.82514 accuracy.
What you can do after
Apply it via a custom boosting implementation to scale gradient updates with curvature and regularization.

Read how Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI explains Hessian matrix →

Also covered in

What to know first

Related concepts

Questions about Hessian matrix

What is Hessian matrix?

A Hessian matrix is the square matrix of second-order partial derivatives of a scalar-valued function of multiple variables.

Which Manning books explain Hessian matrix?

Ensemble Methods for Machine Learning, Quantization and Fast Inference and Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI.

What should I know before learning Hessian matrix?

Manning books that teach Hessian matrix usually introduce gradient boosting, gradient descent and overfitting first.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.