Mutation modifies an existing tensor directly instead of creating a separate result tensor. In PyTorch, a trailing underscore usually marks an in-place method, such as zero_. section 3.8.2
Distinguish mutation from ordinary tensor operations:
- zero_ changes every element of the input tensor, while a method without the underscore leaves the source unchanged and returns a new tensor. section 3.8.2
- Use mutation carefully with autograd because changing values can interfere with values needed during backward computation. section 5.5.1
- Update parameters in place inside a no_grad context when implementing the demonstrated training loop. section 5.5.1
Zeroing a tensor and its gradient
a = torch.ones(3, 2) followed by a.zero_() changes a to a tensor of zeros. During training, params.grad.zero_() clears accumulated gradients before the next iteration. section 3.8.2 section 5.5.1
# In[73]:
a = torch.ones(3, 2)
# In[74]:
a.zero_()
a
# Out[74]:
tensor([[0., 0.],
[0., 0.],
[0., 0.]])
Inspect a runnable example showing an existing tensor transformed directly by zero_. section 3.8.2
