AdaGrad

As explained in Math and Architectures of Deep Learning

An adaptive gradient optimization method that decreases the effective learning rate for parameters that have received large historical gradients.

AdaGrad adapts the learning rate separately for each parameter dimension. This addresses asymmetric loss surfaces, where one shared learning rate may make steep dimensions oscillate while leaving shallow dimensions slow to progress. section 9.2.7 section 9.0

It maintains a state vector containing the cumulative sum of squared partial derivatives for every dimension. Historically large gradients produce larger accumulated values and therefore smaller effective learning rates; historically small gradients retain larger effective learning rates. A small ε prevents division by zero, after which weights and biases receive the adjusted update. section 9.2.7

A speaking-time analogy

AdaGrad gives less attention to dimensions that have produced large slopes repeatedly and more attention to dimensions that have changed infrequently—like paying less attention to someone who speaks constantly and more to someone who rarely speaks. section 9.2.7

Equation 9.14

Use the equation figure to inspect AdaGrad’s accumulated-gradient state and per-dimension update. section 9.2.7

How Math and Architectures of Deep Learning covers it

How deep it goes

Derives the squared-gradient accumulator, unrolled history, ε adjustment, and weight-and-bias updates.

Examples

Uses steep-versus-shallow loss-surface dimensions and a speaking-frequency analogy to explain adaptive emphasis.

What it assumes

Assumes gradients, partial derivatives, parameter vectors, learning rates, and iterative loss minimization.

What you can do after

Evaluate how historical gradient magnitudes alter each parameter’s effective learning rate.

How it connects

Connects loss surfaces and gradient descent with momentum, Nesterov, Adam, and optimization.

liveBookmanning.com

Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthAdaGrad is an adaptive optimization method that scales each parameter's learning rate using the accumulated squared gradients for that parameter.

Compare how 2 books explain AdaGrad →

Related concepts

• epsuses · Eps is a numeric parameter or constant whose specific meaning depends on the method, often representing a tolerance, smoothing value, neighborhood radius, or floating-point precision.
• learning rateconfigures · A learning rate is a step-size parameter that controls the magnitude of updates made by an iterative learning or optimization process.
• momentumuses · Momentum is an optimization technique that incorporates a running history of gradients or parameter updates to smooth and accelerate movement during training.
• gradientuses · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
• RMSpropalternative to · RMSprop is an adaptive gradient-based optimization algorithm that scales parameter updates using an exponentially decaying average of recent squared gradients.

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.