liveBook · Manning

Adam

As explained in Building Specialized AI Systems

Adam is an adaptive gradient-based optimization algorithm that maintains per-parameter estimates of first and second gradient moments.

Adam is an adaptive optimizer that adjusts the learning rate separately for each parameter. It gives larger steps to some parameters and smaller steps to others during fine-tuning. section 3.1.2

Position Adam among common optimization choices:

  • SGD estimates gradients from small random batches, trading exactness for faster updates and useful exploratory noise. section 3.1.2
  • Adam adapts step sizes per parameter, while AdamW adds improved weight decay to help prevent overfitting. section 3.1.2
  • Adafactor targets memory efficiency for truly massive models, making optimizer selection partly a resource trade-off. section 3.1.2

Adaptive hiking assistant

Think of Adam as a hiking assistant that remembers which terrain is steep or gentle and adjusts the step size for each path. section 3.1.2

How Building Specialized AI Systems covers it

How deep it goes
Defines per-parameter learning-rate adaptation and contrasts Adam with SGD, AdamW, and Adafactor without detailing optimizer internals.
Examples
Uses a hiking-assistant analogy contrasting steep and gentle terrain to illustrate parameter-specific step sizes.
What it assumes
Assumes gradient descent, loss, backpropagation, parameters, and minibatch training as the surrounding fine-tuning foundations.
What you can do after
Evaluate optimizer choices against update behavior, model scale, memory demands, and overfitting concerns.
How it connects
Connects Adam to gradient descent, stochastic gradient descent, learning rates, AdamW, Adafactor, weight decay, and overfitting.

Also explained in

Compare how 10 books explain adam →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.