Newton-raphson method
The Newton-Raphson method is an iterative numerical technique that uses first- and second-derivative information to find a root or optimize an objective.
Ensemble Methods for Machine Learning
Newton's method is an iterative optimization method that uses first- and second-derivative information to choose updates toward a local minimum.
- How it is used
- Motivates Newton boosting by showing how curvature-aware minimization trains models and approximates sequential ensemble updates.
- How deep it goes
- Derives gradient-and-Hessian updates, gives convergence pseudocode, and implements optimization for Branin and squared-loss classification.
- Examples
- Uses the Branin function, a saddle point, and a synthetic two-class squared-loss linear classifier.
- What you can do after
- Build it from scratch by computing gradients and Hessians, applying Newton updates, and stopping when parameter changes meet a tolerance.
Read how Ensemble Methods for Machine Learning explains newton-raphson method →
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
A Newton-Raphson update is an optimization step that uses first- and second-order derivatives to refine parameter values.
- How it is used
- Uses second-order optimization to update predictions within gradient boosting and XGBoost.
- How deep it goes
- Explains gradients, Hessians, curvature, adaptive learning rates, regularization, and a custom implementation, but avoids a full derivation.
- Examples
- Demonstrates a NewtonianGradientBoosting class that fits decision trees using gradient, Hessian, and regularization values.
- What you can do after
- Apply it via XGBoost or implement a Newton-style boosting update with Hessian-based regularization.
Related concepts
- gradient descentcontrasts with · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
- Hessian matrixuses · A Hessian matrix is the square matrix of second-order partial derivatives of a scalar-valued multivariable function, describing its local curvature.
- Newton boostingused by · Newton boosting is a boosting approach that uses first- and second-order loss information to construct successive learners.
- second-order optimizationkind of · Second-order optimization is an optimization approach that uses first-derivative information together with curvature information, such as a Hessian or its approximation.
- quadratic approximationused by · A quadratic approximation estimates a function locally using terms through second order, including its value, first derivative, and second derivative.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.