A training loop repeatedly processes batches, computes an error, and updates trainable parameters. In knowledge distillation, it compares a student model with a teacher to recover capabilities lost through pruning. section 2.5 section 6.3.2
A distillation iteration can be organized into five stages:
- Load a batch of token IDs and attention masks onto the training device. A PyTorch DataLoader can shuffle examples and group them into batches. section 2.5
- Run the teacher forward pass under torch.no_grad() and the student forward pass with gradients enabled. Put the teacher in evaluation mode and the student in training mode. section 2.5 section 6.3.2
- Compute a loss comparing the outputs. A logits-only approach uses KL divergence; a compound loss can also compare task outputs and hidden representations. section 2.5 section 6.3.2
- Backpropagate the loss, step an optimizer configured for the student’s parameters, and clear gradients before the next update. section 2.5 section 6.3.2
- For hidden-state alignment, request the model’s hidden states and exclude the first tuple element, which is the embedding output rather than a transformer block output. section 6.3.2
Listing 2.5 Step 4: Correcting the attempt
loss = F.kl_div( F.log_softmax(student_logits, dim=-1), F.softmax(teacher_logits, dim=-1), reduction='batchmean' ) loss.backward() #A optimizer.step() #B optimizer.zero_grad() #C
This update step compares student and teacher logits with KL divergence, then backpropagates, updates parameters, and clears gradients. section 2.5
Comparing recovery strategies
At the 15K data scale, the pruned baseline has perplexity 121.84 and capability retention 78.6, versus 12.65 and 90.1 for labels-only distillation. The teacher baseline has perplexity 12.96 and retention 100, so lower perplexity alone does not indicate complete capability recovery. section 6.3.2
Table 6.3 Comparing Knowledge Distillation strategies
|
Experiment
|
Techniques
|
Map
|
Perplexity
2K / 15K / 40K
|
Capabilities
Retention
2K / 15K / 40K
|
LAMBADA
2K / 15K / 40K
|
|
Baseline
|
Teacher
|
—
|
13.38 / 12.96 / 12.91
|
100 / 100 / 100
|
0.43 / 0.43 / 0.43
|
|
Baseline
|
Pruned
|
—
|
125.60 / 121.84 / 120.67
|
78.6 / 78.6 / 78.30
|
0.177 / 0.177 / 0.177
|
|
Labels-Only
|
Soft & Hard Labels KD
|
—
|
18.86 / 12.65 / 11.73
|
86.74 / 90.1 / 90.1
|
0.295 / 0.332 / 0.335
|
|
Features
|
Feature Alignment
|
Last Layer
|
18.85 / 12.75 / —
|
86.3 / 89.6 / —
|
0.293 / 0.330 / —
|
|
Advanced KD
|
Feature Alignment + FDD + SKD
|
Selected Layer
|
18.94 / 12.54 / 11.54
|
86 / 89.2 / 90
|
0.294 / 0.320 / 0.331
|
|
Advanced KD
|
Feature Alignment
+ FDD + SKD
|
Uniform
|
18.96 / 12.56
/ —
|
86.3 / 89.5 / —
|
0.295 / 0.327 / —
|
Compare the teacher and pruned baselines with labels-only, feature-alignment, and advanced distillation across data scales and metrics. section 6.3.2






