liveBook · Manning

Training loop

As explained in Build a DeepSeek Model (From Scratch)

A training loop is an iterative procedure that processes batches, computes predictions and loss, backpropagates gradients, updates parameters, and clears gradients.

A training loop is the repeated procedure that turns batches into parameter updates. It fetches data, computes predictions and loss, backpropagates gradients, updates weights, and clears gradients before the next batch. section 6.4.1

A single optimization step follows this sequence:

  • Fetch a mini-batch of inputs and targets. section 6.4.1
  • Run the forward pass and calculate the loss against the targets. section 6.4.1
  • Call loss.backward() to calculate gradients for the model weights. section 6.4.1
  • Call optimizer.step() to update the weights, then optimizer.zero_grad() to prevent stale gradients from affecting the next step. section 6.4.1

A complete loop also schedules the learning rate, switches between training and evaluation modes, disables gradients during validation, and measures validation loss periodically. Checkpointing preserves model and optimizer state so interrupted runs can resume. section 6.3.3 section 6.4.1

PyTorch language-model loop

For each iteration, set the scheduled learning rate, call model.train(), fetch a training batch, run the model inside mixed-precision autocast, backpropagate its loss, and update the optimizer. At evaluation intervals, call model.eval(), compute several validation losses inside torch.no_grad(), and save both the best and periodic checkpoints. section 6.3.3 section 6.4.1

Listing 6.23 The training and evaluation loop
# 3. Training Loop
t0 = time.time()
best_val_loss = float('inf')
for iter_num in range(max_iters):
    # Update learning rate
    lr = get_lr(iter_num)
    for param_group in optimizer.param_groups: param_group['lr'] = lr
    # --- Training Step —
    model.train() #A
    X, Y = get_batch('train')
    with ctx:
        outputs = model(X, targets=Y) #B
        loss = outputs['loss']
    optimizer.zero_grad(set_to_none=True)
    loss.backward() #C
    optimizer.step()
    # --- Evaluation Step ---
    if iter_num % eval_interval == 0 and iter_num > 0:
        model.eval() #D
        losses = torch.zeros(eval_iters)
        with torch.no_grad(): #E
            for k in range(eval_iters):
                X, Y = get_batch('val')
                with ctx:
                    outputs = model(X, targets=Y)
                    losses[k] = outputs['loss'].item()
        val_loss = losses.mean()
        print(f"iter {iter_num}: validation loss {val_loss:.4f}")
        # ... (checkpointing logic comes next) …

Trace the training, evaluation, learning-rate, mixed-precision, gradient, and optimizer operations in a runnable loop. section 6.3.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains the complete update sequence, evaluation, scheduling, and checkpointing, but stops short of distributed training.
Examples
Uses a PyTorch language-model loop with TinyStories batches, validation loss, mixed precision, and best or periodic checkpoint files.
What it assumes
Assumes a prepared dataset, defined model, optimizer, loss-producing forward path, and configured device.
What you can do after
Build it from scratch by combining batches, model calls, loss computation, backpropagation, optimizer updates, evaluation, and checkpointing.
How it connects
Connects mini-batches, forward passes, loss functions, backpropagation, optimizers, learning-rate schedules, validation, mixed precision, and checkpointing.

Also explained in

Compare how 7 books explain training loop →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.