AdaGrad
AdaGrad is an adaptive optimization algorithm that adjusts each parameter’s learning rate using the historical accumulation of its squared gradients.
Math and Architectures of Deep Learning
An adaptive gradient optimization method that decreases the effective learning rate for parameters that have received large historical gradients.
- How it is used
- Adapts learning rates per parameter dimension during neural-network loss optimization.
- How deep it goes
- Derives the squared-gradient accumulator, unrolled history, ε adjustment, and weight-and-bias updates.
- Examples
- Uses steep-versus-shallow loss-surface dimensions and a speaking-frequency analogy to explain adaptive emphasis.
- What you can do after
- Evaluate how historical gradient magnitudes alter each parameter’s effective learning rate.
Read how Math and Architectures of Deep Learning explains AdaGrad →
Machine Learning Algorithms in Depth
AdaGrad is an adaptive optimization method that scales each parameter's learning rate using the accumulated squared gradients for that parameter.
- How it is used
- Deploys AdaGrad as a neural-network optimizer when gradient magnitudes differ across objective directions.
- How deep it goes
- Derives the motivation and parameter-wise scaling mechanism, then contrasts cumulative history with RMSProp's moving average.
- Examples
- Uses slowly versus rapidly varying log likelihoods and optimizer comparisons for LeNet trained on CIFAR-100.
Read how Machine Learning Algorithms in Depth explains AdaGrad →
What to know first
- chain ruleThe chain rule is a calculus rule for differentiating a composite function by combining the derivatives of its component functions.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- objective functionAn objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
Related concepts
- epsuses · Epsilon (eps) is a numerical parameter or constant, commonly a small positive value, that may represent a tolerance, distance threshold, stabilizer, or floating-point precision.
- gradientuses · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- learning rateconfigures · A learning rate is a numerical step-size parameter that controls the magnitude of successive updates to model parameters or estimates during iterative learning or optimization.
- momentumuses · Momentum is an optimization technique that incorporates a running history of gradients or parameter updates to smooth and accelerate movement during training.
- RMSpropalternative to · RMSprop is an adaptive gradient-based optimization algorithm that scales parameter updates using an exponentially decaying average of recent squared gradients.
Questions about AdaGrad
What is AdaGrad?
AdaGrad is an adaptive optimization algorithm that adjusts each parameter’s learning rate using the historical accumulation of its squared gradients.
Which Manning books explain AdaGrad?
Math and Architectures of Deep Learning and Machine Learning Algorithms in Depth.
What should I know before learning AdaGrad?
Manning books that teach AdaGrad usually introduce chain rule, language model and objective function first.
What is the difference between AdaGrad and RMSprop?
AdaGrad is an adaptive optimization algorithm that adjusts each parameter’s learning rate using the historical accumulation of its squared gradients. RMSprop is an adaptive gradient-based optimization algorithm that scales parameter updates using an exponentially decaying average of recent squared gradients.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

