Hessian matrix
A Hessian matrix is the square matrix of second-order partial derivatives of a scalar-valued function of multiple variables.
Ensemble Methods for Machine Learning
A Hessian is the square matrix of second-order partial derivatives of a scalar-valued function.
- How it is used
- Guides Newton’s method and Newton boosting with second-order curvature information.
- How deep it goes
- Explains multivariate construction, inverse-Hessian updates, analytic and numerical derivatives, and leaf-node use without detailing large-problem handling.
- Examples
- Uses a three-variable function, the Branin function, focal-loss derivatives, and LightGBM training on breast-cancer data.
- What you can do after
- Apply it via LightGBM custom objectives or implement Newton updates from gradients and Hessians.
Read how Ensemble Methods for Machine Learning explains Hessian matrix →
Quantization and Fast Inference
A Hessian is a matrix of second-order partial derivatives describing the local curvature of a scalar-valued function.
- How it is used
- Uses activation-dependent sensitivity to guide weight rounding during GPTQ quantization.
- How deep it goes
- Defines the Hessian through layer reconstruction loss, gives H = XXᵀ, and connects its inverse to error compensation.
- Examples
- Uses OPT-6.7B sensitivity outliers and compares GPTQ with round-to-nearest quantization at 3-bit and 4-bit.
- What you can do after
- Apply it via GPTQ to guide weight quantization using calibration activations and compensate rounding errors.
Read how Quantization and Fast Inference explains Hessian matrix →
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
The square matrix of second-order partial derivatives of a scalar multivariable function, describing its local curvature.
- How it is used
- Uses second-order curvature information to adapt gradient-boosting updates and accelerate Newton-style optimization.
- How deep it goes
- Explains curvature, update behavior, regularization, and implementation, but defers detailed analytical and numerical derivations.
- Examples
- Demonstrates Hessian-weighted updates in a custom NewtonianGradientBoosting class and reports 0.82514 accuracy.
- What you can do after
- Apply it via a custom boosting implementation to scale gradient updates with curvature and regularization.
Read how Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI explains Hessian matrix →
Also covered in
What to know first
- gradient boostingGradient boosting is an ensemble-learning technique that sequentially adds predictive models fitted to reduce a loss by following its gradient.
- gradient descentGradient descent is an iterative optimization method that updates parameters in the direction opposite an objective function’s gradient to reduce the objective’s value.
- overfittingOverfitting is a modeling condition in which a model adapts too closely to particular data, including its noise or idiosyncrasies, and consequently generalizes poorly to new data.
Related concepts
- GPTQused by · GPTQ is a post-training weight-quantization method that uses calibration data and approximate second-order information to reduce neural-network precision while limiting quantization error.
- newton-raphson methodused by · The Newton-Raphson method is an iterative numerical technique that uses first- and second-derivative information to find a root or optimize an objective.
- XGBoostused by · XGBoost is an open-source software library for efficient, scalable gradient-boosted decision-tree algorithms.
- Jacobianequivalent to · The matrix of all first-order partial derivatives of a vector-valued function with respect to its inputs.
- Taylor seriesused by · A Taylor series is a power series whose coefficients are determined by a function’s derivatives evaluated at a specified expansion point.
- Newton boostinguses · Newton boosting is a boosting approach that uses first- and second-order loss information to construct successive learners.
Questions about Hessian matrix
What is Hessian matrix?
A Hessian matrix is the square matrix of second-order partial derivatives of a scalar-valued function of multiple variables.
Which Manning books explain Hessian matrix?
Ensemble Methods for Machine Learning, Quantization and Fast Inference and Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI.
What should I know before learning Hessian matrix?
Manning books that teach Hessian matrix usually introduce gradient boosting, gradient descent and overfitting first.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


