Use a PyTorch tensor’s .grad attribute to retrieve the gradient produced by backpropagation. After loss.backward(), PyTorch stores the loss gradient on every involved tensor that tracks gradients, such as model weights, biases, or an input image. section 3.4.2 section 3.4.1
Read, use, and clear parameter gradients
A manual training step follows a reliable sequence: compute predictions and loss, call loss.backward(), read W.grad and b.grad, update parameters, then set W.grad and b.grad to None. The update belongs inside torch.no_grad() so it does not become part of the backward computation. section 3.4.2
input_dim = 2 output_dim = 1 W = torch.rand(input_dim, output_dim, requires_grad=True) b = torch.zeros(output_dim, requires_grad=True)
These model variables explicitly track gradients, making their .grad attributes available after backpropagation. section 3.4.2
Keep these practical rules in mind:
- Set requires_grad=True when creating tensors whose gradients you need. section 3.4.2
- Expect repeated backward() calls to accumulate gradients rather than replace them; reset .grad to None between training steps. section 3.4.1
- When a tensor is recreated every iteration, as with an optimized image, its gradients need not be reset separately. section 10.2.2
