Adam is an adaptive optimizer that adjusts the learning rate separately for each parameter. It gives larger steps to some parameters and smaller steps to others during fine-tuning. section 3.1.2
Position Adam among common optimization choices:
- SGD estimates gradients from small random batches, trading exactness for faster updates and useful exploratory noise. section 3.1.2
- Adam adapts step sizes per parameter, while AdamW adds improved weight decay to help prevent overfitting. section 3.1.2
- Adafactor targets memory efficiency for truly massive models, making optimizer selection partly a resource trade-off. section 3.1.2
Adaptive hiking assistant
Think of Adam as a hiking assistant that remembers which terrain is steep or gentle and adjusts the step size for each path. section 3.1.2









