liveBook · Manning

Robbins-Monro conditions

As explained in Machine Learning Algorithms in Depth

The Robbins-Monro conditions are convergence requirements on a sequence of stochastic-approximation step sizes, typically involving divergent sums of step sizes and convergent sums of their squares.

Robbins–Monro conditions constrain stochastic-approximation step sizes so updates can continue moving toward a solution without sustaining uncontrolled oscillations. section 5.4 section 10.4

A step-size sequence satisfies the standard conditions when it has both properties:

  • The sum of step sizes diverges, allowing the algorithm to make enough cumulative progress to find a locally optimal solution. section 5.4 section 10.4
  • The sum of squared step sizes converges, controlling oscillations as stochastic updates accumulate. section 5.4 section 10.4

Power-law learning-rate schedule

The schedule ηk = (τ0 + k)^−κ satisfies the conditions when τ0 ≥ 0 and κ ∈ (0.5, 1]. The offset τ0 slows early iterations, while κ controls how quickly older updates are forgotten. section 5.7 section 5.4

(5.27)
figure

Use the displayed learning-rate equation to inspect the schedule used for stochastic-gradient updates. section 5.4

How Machine Learning Algorithms in Depth covers it

How deep it goes
Defines divergent step-size sums and convergent squared sums, then verifies a power-law schedule without developing convergence proofs.
Examples
Uses ηk = (τ0 + k)^−κ and applies it in logistic-regression SGD and neural-network learning-rate scheduling.
What you can do after
Operate/configure learning-rate schedules that meet standard stochastic-gradient convergence requirements.
How it connects
Links stochastic gradient descent, learning-rate schedules, logistic regression, and neural-network optimizers.
Cover of Machine Learning Algorithms in Depth
liveBookmanning.com
Read Machine Learning Algorithms in Depth on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Algorithms in Depth. The section links point to the parts of the book it draws on.