Training needs gradients to update model parameters. Validation does not: it measures performance without changing the weights. Wrap a validation pass in `with torch.no_grad():` to tell PyTorch not to build an autograd graph for those calculations. Avoiding that work can save time and memory, especially with a large model. section 5.5.5 section 13.5.2
A validation pass
After computing the training loss normally, calculate the validation loss inside the context: `with torch.no_grad():` ` val_prediction = model(val_input, *params)` ` val_loss = loss_fn(val_prediction, val_target)` There is no validation backward pass or parameter update. In this example, `val_loss.requires_grad` is `False`. The `with` statement confines the change in gradient tracking to its block. section 5.5.5 section 13.5.2
`no_grad()` is also useful *during* a hand-written training loop, but only around the parameter update. First compute the loss and call `backward()` with autograd active; then update the parameters inside `with torch.no_grad():`. This prevents the update from becoming part of the next forward graph and permits the in-place update shown in the example. Gradients still need to be cleared between iterations when they should not accumulate. section 5.5.1 section 5.8
# In[9]:
def training_loop(n_epochs, learning_rate, params, t_u, t_c):
for epoch in range(1, n_epochs + 1):
if params.grad is not None: #1
params.grad.zero_()
t_p = model(t_u, *params)
loss = loss_fn(t_p, t_c)
loss.backward()
with torch.no_grad(): #2
params -= learning_rate * params.grad
if epoch % 500 == 0:
print('Epoch %d, Loss %f' % (epoch, float(loss)))
return params
A complete small training loop places `backward()` before `no_grad()` and keeps only the in-place parameter update inside the context. section 5.5.1
In larger validation loops, `model.eval()` appears alongside `torch.no_grad()`: the model is put in evaluation mode, while gradient computation is disabled. The loop can still collect losses or other metrics without using them to update weights. If you specifically need a tensor guaranteed to be detached, use `detach()` instead of assuming every output created in a `no_grad` block has `requires_grad=False`; views can be an exception. To switch tracking according to a training/inference Boolean, use the related `torch.set_grad_enabled(is_train)` context. section 13.5.2 section 5.5.5
Listing 13.14 Validation loop using eval() mode and no_grad context
def main(self):
for epoch_ndx in range(1, self.cli_args.epochs + 1):
# ... line 157
valMetrics_t = self.doValidation(epoch_ndx, val_dl)
self.logMetrics(epoch_ndx, 'val', valMetrics_t)
# ... line 203
def doValidation(self, epoch_ndx, val_dl):
with torch.no_grad():
self.model.eval() #1
valMetrics_g = torch.zeros(
METRICS_SIZE,
len(val_dl.dataset),
device=self.device,
)
val_progress = tqdm(
val_dl,
desc="E{} Validation".format(epoch_ndx),
total=len(val_dl)
)
for batch_ndx, batch_tup in enumerate(val_progress):
self.computeBatchLoss(
batch_ndx, batch_tup, val_dl.batch_size, valMetrics_g)
return valMetrics_g.to('cpu')
A classification validation loop combines `torch.no_grad()`, `model.eval()`, and per-batch metric collection without an optimizer step. section 13.5.2
