Adam is an optimizer that adjusts model parameters to reduce a loss. Unlike basic gradient descent, which considers the current gradient alone, Adam also uses information from earlier gradients. It is a practical choice for training neural networks without designing an update rule from scratch. section 2.2.1 section 2.3.2
Use Adam in a classification model
For a binary image classifier, create `optimizer = torch.optim.Adam(binary_model.parameters(), lr=0.001)` alongside binary cross-entropy loss. For each batch, make predictions, calculate loss, then call `optimizer.zero_grad()`, `loss.backward()`, and `optimizer.step()`. The backward call computes gradients; the step call updates the parameters. section 2.3.2
Adjust the setup to the task:
- Learning rate (`lr`) controls the size of weight adjustments. It can be selected using a validation set; examples use 0.001 for classification and 0.0001 for a music Transformer. section 2.2.1 section 14.4.1
- A GAN uses separate Adam optimizers for its generator and discriminator. A DCGAN example sets `lr=0.0002` and `betas=(0.5, 0.999)` for both; the betas govern how recent and past gradient information influences training. section 3.3.3 section 4.5.1
- A music Transformer clips the gradient norm to 1 after `loss.backward()` and before `optimizer.step()` to guard against exploding gradients. section 14.4.1
Listing 2.6 Training a binary classification model
for i in range(50): #1
tloss=0
for n,(imgs,labels) in enumerate(train_loader): #2
imgs=imgs.reshape(-1,28*28) #3
imgs=imgs.to(device)
labels=torch.FloatTensor(\
[x if x==0 else 1 for x in labels]) #4
labels=labels.reshape(-1,1).to(device)
preds=binary_model(imgs)
loss=loss_fn(preds,labels) #5
optimizer.zero_grad()
loss.backward() #6
optimizer.step()
tloss+=loss.detach()
tloss=tloss/n
print(f"at epoch {i}, loss is {tloss}")
Inspect the complete binary-classification loop to see where gradient clearing, backpropagation, and Adam's parameter update occur within each batch. section 2.3.2
