Optimization turns computed gradients into parameter updates that reduce the training loss. In a PyTorch training script, the optimizer is configured after model initialization, receives the model parameters, and updates them after each backward pass. section 6.3.3
A practical optimizer step follows this sequence:
- Fetch a mini-batch and run the model to calculate the loss. section 6.4.1
- Call loss.backward() to calculate gradients for every model weight. section 6.4.1
- Call optimizer.step() to nudge the weights in the direction that reduces loss, then clear gradients with optimizer.zero_grad(). section 6.4.1
- Update the learning rate from a scheduler before the next iteration. section 6.3.3
Training-loop update
A typical iteration runs the model under mixed precision, obtains the combined loss, clears old gradients, computes new gradients with loss.backward(), and updates the weights with optimizer.step(). The optimizer state is saved alongside model weights in checkpoints so training can resume.





































