How deeply does Machine Learning Algorithms in Depth cover AdaGrad?

Derives the motivation and parameter-wise scaling mechanism, then contrasts cumulative history with RMSProp's moving average.

AdaGrad

As explained in Machine Learning Algorithms in Depth

AdaGrad is an adaptive optimization method that scales each parameter's learning rate using the accumulated squared gradients for that parameter.

AdaGrad is an adaptive optimization method that adjusts each parameter's effective learning rate using the squared gradients accumulated for that parameter. A fixed learning rate struggles with both slowly varying objectives, where larger steps help, and rapidly varying objectives, where large steps cause oscillation. section 10.4

AdaGrad proceeds by accumulating gradient history and scaling updates:

Compute the gradient for each parameter at the current optimization step. section 10.4
Accumulate the squared gradients seen so far separately for each parameter. section 10.4
Divide the learning rate by the square root of each parameter's accumulated squared-gradient sum. section 10.4
Reduce effective learning rates for parameters receiving large gradients while preserving relatively larger steps in gently sloped directions. section 10.4

Contrasting objective landscapes

For a slowly varying objective, small gradients make a larger effective learning rate useful. For a rapidly varying objective, large gradients reduce the effective learning rate and limit oscillation around the optimum. section 10.4

AdaGrad's central trade-off is cumulative: its learning rates decrease as more gradient history is collected. RMSProp modifies this behavior by replacing the full accumulation with an exponentially weighted moving average that discards distant history. section 10.4

How Machine Learning Algorithms in Depth covers it

Cover of Machine Learning Algorithms in Depth

liveBook manning.com

Read Machine Learning Algorithms in Depth on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Math and Architectures of Deep LearningAn adaptive gradient optimization method that decreases the effective learning rate for parameters that have received large historical gradients.

Compare how 2 books explain AdaGrad →

Related concepts

epsuses · Eps is a numeric parameter or constant whose specific meaning depends on the method, often representing a tolerance, smoothing value, neighborhood radius, or floating-point precision.
learning rateconfigures · A learning rate is a step-size parameter that controls the magnitude of updates made by an iterative learning or optimization process.
momentumuses · Momentum is an optimization technique that incorporates a running history of gradients or parameter updates to smooth and accelerate movement during training.
gradientuses · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
RMSpropalternative to · RMSprop is an adaptive gradient-based optimization algorithm that scales parameter updates using an exponentially decaying average of recent squared gradients.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Algorithms in Depth. The section links point to the parts of the book it draws on.