An optimizer updates model parameters to reduce a loss. In PyTorch, it receives the parameters to change, uses their gradients, and applies an update controlled by a learning rate. SGD is the simplest example: gradient descent moves parameters toward lower loss, while the optimizer packages the repeated update operation. section 1.4 section 5.1.2
Use an optimizer inside the standard training loop:
- Create it with the parameters to update, for example torch.optim.SGD(model.parameters(), lr=0.001). section 2.1.3 section 5.1.3
- For each batch, call optimizer.zero_grad(), compute predictions and loss, call loss.backward(), then call optimizer.step(). section 1.4 section 2.1.3
- Treat the learning rate as the step-size control: gradients provide the direction, while the optimizer can transform them to handle noisy, exploding, or vanishing gradients. section 5.1.2
- Replace SGD with another optimizer through the same interface when the task benefits from a different update rule; use a learning-rate scheduler separately when needed. section 5.1.3
Minimizing a one-variable function
A parameter initialized at x = 3.5 can minimize f(x) = (x − 2)² with SGD. The loop clears old gradients, evaluates the function, backpropagates, and steps 60 times; the parameter moves toward 2. section 1.4
Figure 2.2 Diagram of the training loop process. It includes the same three major steps of training a model in PyTorch that we originally described, with matching color-coding. New is the initialization of objects that are reused in every training loop. Solid areas show steps, and dashed arrows show effects.
Use the training-loop diagram to connect optimizer creation with batch updates, loss computation, backpropagation, and parameter changes. section 2.1.3
