Second derivative

As explained in Ensemble Methods for Machine Learning

A second derivative is the derivative of a function's first derivative and describes its local curvature with respect to a variable.

A second derivative describes how a function’s slope changes. In optimization, it adds local curvature to the first derivative, replacing a merely linear approximation with a quadratic one and often producing more informative steps. section 6.1

Use the second derivative to:

• Differentiate a univariate function twice to obtain its second derivative. section 6.1
• Represent second-order information for multivariate functions with a Hessian matrix. section 6.1
• Construct Newton updates from the gradient and inverse Hessian, typically using a unit step length. section 6.1.1
• Weight boosting residuals through the Hessian, focusing learning more strongly on badly misclassified examples. section 6.2.1

Newton’s descent on the Branin function

Starting from [2, −5], compute the Branin function’s gradient and Hessian, set the descent direction to −H⁻¹g, update with step length 1, and stop when successive estimates are sufficiently close. Compared with gradient descent, the resulting path reaches a local minimum more directly, although both methods can converge to a saddle point. section 6.1.1

Figure 6.1 Comparing gradient descent (left) and Newton’s method (right). Gradient descent only uses local first-order information near the current solution, which leads to a linear approximation of the function being optimized. Different step lengths will then lead to different next steps. Newton’s method uses both local first- and second-order information near the current solution, leading to a quadratic (parabolic) approximation of the function being optimized. This provides a better estimate of the next step.

Compare first-order linear and second-order quadratic approximations visually; the contrast clarifies why curvature can improve optimization steps. section 6.1

How Ensemble Methods for Machine Learning covers it

How deep it goes

Explains repeated differentiation, Hessian representation, quadratic approximation, and Newton updates with an implemented example.

Examples

Uses the Branin function, logistic loss, and Hessian-weighted residuals to illustrate curvature and optimization.

What it assumes

Assumes familiarity with first derivatives, gradients, optimization updates, and basic univariate and multivariate functions.

What you can do after

Build it from scratch by computing gradients, Hessians, Newton directions, updates, and convergence checks.

How it connects

Connects second derivatives to first derivatives, Hessian matrices, gradient descent, Newton’s method, and gradient boosting.

liveBookmanning.com

Read Ensemble Methods for Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Geometry for ProgrammersA second derivative is the derivative of a first derivative and measures the rate of change of a function's slope or tangent direction.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA second derivative measures how the rate of change of a function itself changes with respect to its input.

Compare how 3 books explain second derivative →

Related concepts

• curvaturemeasures · Curvature is a measure of how a curve or surface bends locally, or, for a function, how its slope changes across its input space.
• derivativekind of · A derivative is a mathematical quantity that measures the instantaneous rate of change of a function with respect to an input variable.
• tangent planemeasures · A tangent plane is the plane that locally best approximates a smooth surface at a specified point.
• convexitymeasures · Convexity is the property of a set containing the line segment between any two of its points, or of a function whose value between two points is no greater than the linear interpolation of its endpoint values.
• Newton boostinguses · Newton boosting is a boosting approach that uses first- and second-order loss information to construct successive learners.
• Taylor seriesused by · A Taylor series is a power series whose coefficients are determined by a function’s derivatives evaluated at a specified expansion point.

This explainer was generated by Manning's knowledge graph from the text of Ensemble Methods for Machine Learning. The section links point to the parts of the book it draws on.