Adam

As explained in Deep Learning with R, Third Edition

Adam is an adaptive stochastic optimization algorithm that uses estimates of first and second moments of gradients to update parameters.

model |> compile(
  optimizer = "adam",
  loss = "binary_crossentropy",
  metrics = c("accuracy")
)

model |> compile(
  optimizer = "adam",
  loss = "sparse_categorical_crossentropy",
  metrics = "accuracy"
)

fit(model, train_images, train_labels, epochs = 5, batch_size = 128)

How Deep Learning with R, Third Edition covers it

How deep it goes

Identifies Adam’s optimizer role and default usefulness but does not derive its moment estimates or update equations.

Examples

Uses MNIST digit classification and binary probability classification as concrete Adam configurations.

What it assumes

Assumes basic familiarity with neural-network layers, losses, weights, gradients, and model training.

What you can do after

Apply Adam via Keras to compile and train classification models.

How it connects

Connects Adam to loss functions, gradient descent, backpropagation, mini-batch training, and model compilation.

liveBookmanning.com

Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsAn adaptive optimization algorithm that combines moment estimates with variance-based gradient normalization.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorch
• 

  liveBookmanning.com
  Bayesian Optimization in ActionAdam is an adaptive stochastic-gradient optimization algorithm that maintains moving estimates of first and second moments of gradients.
• 

  liveBookmanning.com
  How Large Language Models WorkAdam is an adaptive stochastic-gradient optimization algorithm that maintains estimates of gradient moments to adjust parameter updates.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningAn adaptive optimization algorithm that combines momentum-like first-moment estimates with per-parameter learning-rate adaptation.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthAdam is an adaptive optimization method that combines momentum-like first-moment estimates with RMSProp-like second-moment scaling and bias correction.

Compare how 7 books explain adam →

Related concepts

• optimizationkind of · Optimization is the systematic selection or adjustment of values, actions, or system designs to improve a specified objective, often subject to constraints.
• stochastic gradient descentkind of · Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly selected small batches.
• momentumuses · Momentum is an optimization technique that incorporates a running history of gradients or parameter updates to smooth and accelerate movement during training.
• AdamWalternative to · AdamW is an adaptive gradient-based optimization algorithm that combines Adam’s moment-based parameter updates with weight decay applied independently of those updates.
• gradient descentimplements · Gradient descent is an iterative optimization method that updates parameters opposite the gradient of an objective function to reduce its value.
• RMSpropuses · RMSprop is an adaptive gradient-based optimization algorithm that scales parameter updates using an exponentially decaying average of recent squared gradients.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.