A step length is the scalar that controls how much an iterative update moves in its chosen direction: next = current + (step length × direction). Changing it can produce very different estimates and can slow convergence. In gradient descent, it scales the negative gradient; in Newton’s method, it is conventionally set to 1 because curvature information supplies an exact step. section 6.1
Its role changes with the optimization or ensemble method:
- Gradient descent computes a step length, commonly written αt, to scale the negative gradient. section 6.1
- Newton’s descent uses the gradient and Hessian to construct a more informative direction, then sets α to 1 instead of computing it. section 6.1.1
- Newton boosting must approximate its step, so its step length becomes the learning rate. section 6.1
- Gradient boosting uses a step length as the hypothesis weight of each weak regressor, typically finding it through line search. section 5.2.1
Newton’s descent on the Branin function
A from-scratch Newton implementation computes the gradient and Hessian, forms d = −H⁻¹g, sets the step length to 1, and updates wnew = wold + αd. Starting at [2, −5], Newton’s path reaches a local minimum more directly than gradient descent because each update uses a quadratic rather than linear local approximation. section 6.1.1 section 6.1
Figure 6.1 Comparing gradient descent (left) and Newton’s method (right). Gradient descent only uses local first-order information near the current solution, which leads to a linear approximation of the function being optimized. Different step lengths will then lead to different next steps. Newton’s method uses both local first- and second-order information near the current solution, leading to a quadratic (parabolic) approximation of the function being optimized. This provides a better estimate of the next step.
Compare how different step lengths affect gradient-descent updates, while Newton’s quadratic approximation provides a better estimate of the next step. section 6.1
