A training loop is the repeated procedure that turns batches into parameter updates. It fetches data, computes predictions and loss, backpropagates gradients, updates weights, and clears gradients before the next batch. section 6.4.1
A single optimization step follows this sequence:
- Fetch a mini-batch of inputs and targets. section 6.4.1
- Run the forward pass and calculate the loss against the targets. section 6.4.1
- Call loss.backward() to calculate gradients for the model weights. section 6.4.1
- Call optimizer.step() to update the weights, then optimizer.zero_grad() to prevent stale gradients from affecting the next step. section 6.4.1
A complete loop also schedules the learning rate, switches between training and evaluation modes, disables gradients during validation, and measures validation loss periodically. Checkpointing preserves model and optimizer state so interrupted runs can resume. section 6.3.3 section 6.4.1
PyTorch language-model loop
For each iteration, set the scheduled learning rate, call model.train(), fetch a training batch, run the model inside mixed-precision autocast, backpropagate its loss, and update the optimizer. At evaluation intervals, call model.eval(), compute several validation losses inside torch.no_grad(), and save both the best and periodic checkpoints. section 6.3.3 section 6.4.1
Listing 6.23 The training and evaluation loop
# 3. Training Loop
t0 = time.time()
best_val_loss = float('inf')
for iter_num in range(max_iters):
# Update learning rate
lr = get_lr(iter_num)
for param_group in optimizer.param_groups: param_group['lr'] = lr
# --- Training Step —
model.train() #A
X, Y = get_batch('train')
with ctx:
outputs = model(X, targets=Y) #B
loss = outputs['loss']
optimizer.zero_grad(set_to_none=True)
loss.backward() #C
optimizer.step()
# --- Evaluation Step ---
if iter_num % eval_interval == 0 and iter_num > 0:
model.eval() #D
losses = torch.zeros(eval_iters)
with torch.no_grad(): #E
for k in range(eval_iters):
X, Y = get_batch('val')
with ctx:
outputs = model(X, targets=Y)
losses[k] = outputs['loss'].item()
val_loss = losses.mean()
print(f"iter {iter_num}: validation loss {val_loss:.4f}")
# ... (checkpointing logic comes next) …Trace the training, evaluation, learning-rate, mixed-precision, gradient, and optimizer operations in a runnable loop. section 6.3.3






