Adam

As explained in How Large Language Models Work

Adam is an adaptive stochastic-gradient optimization algorithm that maintains estimates of gradient moments to adjust parameter updates.

Adam is an adaptive form of stochastic gradient descent. It keeps momentum from earlier updates, helping parameters move faster toward lower loss and sometimes pass a small local minimum. section 4.1.2

Adam trades training memory for potentially more effective optimization:

• It can minimize the loss function faster and reduce the chance of getting stuck in a local minimum. section 4.1.2
• It stores momentum information for each parameter, raising memory requirements to about three times those of plain SGD. section 4.1.2
• That extra information can be discarded after training, but the larger training system is still required beforehand. section 4.1.2

A ball rolling down a loss curve

Imagine gradient descent as a ball moving downhill. Adam gives the ball momentum: repeated movement in one direction makes it roll faster and may carry it past a shallow dip instead of leaving it at a local minimum. section 4.1.2

Figure 4.5 The gradient descent algorithm takes steps to adjust parameters to find the optimal outcome with the least loss. Unfortunately, the algorithm gets stuck in a local minimum, an area of the graph that is not optimal because other parameter values correspond to areas with a lower loss.

See how ordinary gradient descent can stop at a local minimum, clarifying the problem Adam’s momentum is intended to help address. section 4.1.2

How How Large Language Models Work covers it

How deep it goes

Explains momentum, local minima, and memory cost through analogy, but stops short of equations or update rules.

Examples

Uses a ball rolling down a loss curve and figures showing parameters, global minima, and local minima.

What it assumes

Builds from gradient descent and stochastic gradient descent before describing Adam’s added momentum.

What you can do after

Evaluate Adam’s optimization benefit against its substantially higher training-memory requirement.

How it connects

Connects Adam with stochastic gradient descent, gradient descent, loss functions, local minima, neural networks, and parameters.

liveBookmanning.com

Read How Large Language Models Work on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsAn adaptive optimization algorithm that combines moment estimates with variance-based gradient normalization.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorch
• 

  liveBookmanning.com
  Bayesian Optimization in ActionAdam is an adaptive stochastic-gradient optimization algorithm that maintains moving estimates of first and second moments of gradients.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionAdam is an adaptive stochastic optimization algorithm that uses estimates of first and second moments of gradients to update parameters.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningAn adaptive optimization algorithm that combines momentum-like first-moment estimates with per-parameter learning-rate adaptation.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthAdam is an adaptive optimization method that combines momentum-like first-moment estimates with RMSProp-like second-moment scaling and bias correction.

Compare how 7 books explain adam →

Related concepts

• optimizationkind of · Optimization is the systematic selection or adjustment of values, actions, or system designs to improve a specified objective, often subject to constraints.
• stochastic gradient descentkind of · Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly selected small batches.
• momentumuses · Momentum is an optimization technique that incorporates a running history of gradients or parameter updates to smooth and accelerate movement during training.
• AdamWalternative to · AdamW is an adaptive gradient-based optimization algorithm that combines Adam’s moment-based parameter updates with weight decay applied independently of those updates.
• gradient descentimplements · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
• RMSpropuses · RMSprop is an adaptive gradient-based optimization algorithm that scales parameter updates using an exponentially decaying average of recent squared gradients.

This explainer was generated by Manning's knowledge graph from the text of How Large Language Models Work. The section links point to the parts of the book it draws on.