liveBook · Manning

Training loop

As explained in Rearchitecting LLMs

A training loop is an iterative procedure that computes model outputs, evaluates an error, and updates trainable parameters over batches and iterations.

A training loop repeatedly processes batches, computes an error, and updates trainable parameters. In knowledge distillation, it compares a student model with a teacher to recover capabilities lost through pruning. section 2.5 section 6.3.2

A distillation iteration can be organized into five stages:

  • Load a batch of token IDs and attention masks onto the training device. A PyTorch DataLoader can shuffle examples and group them into batches. section 2.5
  • Run the teacher forward pass under torch.no_grad() and the student forward pass with gradients enabled. Put the teacher in evaluation mode and the student in training mode. section 2.5 section 6.3.2
  • Compute a loss comparing the outputs. A logits-only approach uses KL divergence; a compound loss can also compare task outputs and hidden representations. section 2.5 section 6.3.2
  • Backpropagate the loss, step an optimizer configured for the student’s parameters, and clear gradients before the next update. section 2.5 section 6.3.2
  • For hidden-state alignment, request the model’s hidden states and exclude the first tuple element, which is the embedding output rather than a transformer block output. section 6.3.2
Listing 2.5 Step 4: Correcting the attempt
loss = F.kl_div(
   F.log_softmax(student_logits, dim=-1),
   F.softmax(teacher_logits, dim=-1),
   reduction='batchmean'
)
loss.backward()      #A
optimizer.step()     #B
optimizer.zero_grad()  #C
 

This update step compares student and teacher logits with KL divergence, then backpropagates, updates parameters, and clears gradients. section 2.5

Comparing recovery strategies

At the 15K data scale, the pruned baseline has perplexity 121.84 and capability retention 78.6, versus 12.65 and 90.1 for labels-only distillation. The teacher baseline has perplexity 12.96 and retention 100, so lower perplexity alone does not indicate complete capability recovery. section 6.3.2

Table 6.3 Comparing Knowledge Distillation strategies
Experiment
Techniques
Map
Perplexity
2K / 15K / 40K
Capabilities
Retention
2K / 15K / 40K
LAMBADA
2K / 15K / 40K
Baseline
Teacher
—
13.38 / 12.96 / 12.91
100 / 100 / 100
0.43 / 0.43 / 0.43
Baseline
Pruned
—
125.60 / 121.84 / 120.67
78.6 / 78.6 / 78.30
0.177 / 0.177 / 0.177
Labels-Only
Soft & Hard Labels KD
—
18.86 / 12.65 / 11.73
86.74 / 90.1 / 90.1
0.295 / 0.332 / 0.335
Features
Feature Alignment
Last Layer
18.85 / 12.75 / —
86.3 / 89.6 / —
0.293 / 0.330 / —
Advanced KD
Feature Alignment + FDD + SKD
Selected Layer
18.94 / 12.54 / 11.54
86 / 89.2 / 90
0.294 / 0.320 / 0.331
Advanced KD
Feature Alignment   + FDD + SKD
Uniform
18.96 / 12.56   / —
86.3 / 89.5 / —
0.295 / 0.327 / —

Compare the teacher and pruned baselines with labels-only, feature-alignment, and advanced distillation across data scales and metrics. section 6.3.2

How Rearchitecting LLMs covers it

How deep it goes
Walks through batching, forward passes, loss calculation, backpropagation, and optimizer updates, then compares recovery strategies.
Examples
Compares labels-only and feature-based distillation against teacher and pruned baselines using perplexity and capability retention at multiple data scales.
What you can do after
Apply it via PyTorch to update a student model from teacher outputs using logits-only or compound distillation losses.
How it connects
Connects knowledge distillation, pruning, KL divergence, hidden-state alignment, and backpropagation.

Also explained in

Compare how 7 books explain training loop →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.