AdaGrad
AdaGrad is an adaptive optimization algorithm that adjusts each parameter’s learning rate using the historical accumulation of its squared gradients.
Math and Architectures of Deep Learning
An adaptive gradient optimization method that decreases the effective learning rate for parameters that have received large historical gradients.
- How it is used
- Adapts learning rates per parameter dimension during neural-network loss optimization.
- How deep it goes
- Derives the squared-gradient accumulator, unrolled history, ε adjustment, and weight-and-bias updates.
- Examples
- Uses steep-versus-shallow loss-surface dimensions and a speaking-frequency analogy to explain adaptive emphasis.
- What you can do after
- Evaluate how historical gradient magnitudes alter each parameter’s effective learning rate.
Read how Math and Architectures of Deep Learning explains AdaGrad →
Machine Learning Algorithms in Depth
AdaGrad is an adaptive optimization method that scales each parameter's learning rate using the accumulated squared gradients for that parameter.
- How it is used
- Deploys AdaGrad as a neural-network optimizer when gradient magnitudes differ across objective directions.
- How deep it goes
- Derives the motivation and parameter-wise scaling mechanism, then contrasts cumulative history with RMSProp's moving average.
- Examples
- Uses slowly versus rapidly varying log likelihoods and optimizer comparisons for LeNet trained on CIFAR-100.
Read how Machine Learning Algorithms in Depth explains AdaGrad →
Related concepts
- epsuses · Eps is a numeric parameter or constant whose specific meaning depends on the method, often representing a tolerance, smoothing value, neighborhood radius, or floating-point precision.
- learning rateconfigures · A learning rate is a step-size parameter that controls the magnitude of updates made by an iterative learning or optimization process.
- momentumuses · Momentum is an optimization technique that incorporates a running history of gradients or parameter updates to smooth and accelerate movement during training.
- gradientuses · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- RMSpropalternative to · RMSprop is an adaptive gradient-based optimization algorithm that scales parameter updates using an exponentially decaying average of recent squared gradients.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.