Knowledge distillation trains a smaller student model to reproduce useful behavior from a larger teacher. After pruning reduces a model’s size, the original model can serve as the teacher while the pruned model recovers lost capabilities. This recovery requires training compute, even when pruning itself is relatively inexpensive. section 1.2 section 2.5
A basic student learns from the teacher’s full prediction distribution, rather than only its highest-probability answer. Matching output probabilities transfers information about alternatives, but does not by itself demonstrate that the models use the same internal reasoning process. section 2.5 section 6.0
A practical recovery workflow:
- Establish a baseline and prune a copy of the teacher. Compare pruning candidates both immediately after removal and after a small recovery run; the least degraded candidate is not necessarily the only one worth testing. section 2.8 section 6.1.3
- Prepare representative recovery data, tokenize it into input IDs and attention masks, and split it for training and validation. Use diverse general data for broad recovery or task-aligned data when specialization is the objective. section 6.1.2 section 2.7
- Freeze the teacher and run both models on each batch. Compute KL divergence between the student’s log-probabilities and the teacher’s probabilities, then backpropagate and update the student. The teacher forward pass needs no gradients. section 2.5 section 6.1.2
- Evaluate against the teacher and the unrecovered student using metrics suited to the deployment task. Check generated text directly as well as benchmark scores, especially for repetition or factual inconsistencies. section 6.1.3 section 8.6
Figure 2.3 In the knowledge distillation cycle, the Teacher model generates its reasoning (logits) for a given text. The Pruned model (our student) does the same. The KL Divergence loss function measures the difference between both "reasonings," and this error is used to adjust the student's weights, making it think more like the Teacher over time.
The cycle shows how teacher and student outputs are compared to update the student’s weights. section 2.5
More elaborate recovery can combine output alignment with a task loss on dataset labels and alignment of intermediate hidden states. If teacher and student have different numbers of blocks, feature alignment also needs a layer-mapping strategy. Loss weights, mapping, pruning choices, and training data should be compared experimentally; gradient accumulation can help fit larger effective batches into limited GPU memory. section 6.0 section 6.3.2
Recovering a pruned Gemma model
An 18-layer gemma-3-270m teacher was used to recover a 16-layer student trained on 15,000 SlimPajama rows. The student retained the pruned model’s 4.16% parameter reduction and 4.61-second inference time versus 5.12 seconds for the teacher. On ARC Easy, its score rose from 0.49 before recovery to 0.54 afterward, compared with 0.59 for the teacher; other benchmark results varied. section 2.5 section 2.6
More data and more complex losses are not guaranteed to improve every metric. In one comparison, perplexity improved substantially from 2,000 to 15,000 samples, while hidden-state alignment did not consistently outperform label-based distillation on capability retention. Measure training cost and task-specific outcomes before choosing a more complex strategy. section 6.3.2
Distilling a tool-call specialist
For prompt-completion examples, concatenate the prompt and completion, retain the full input, and set prompt labels to -100 so they do not contribute to the label loss. Mask padding labels similarly. A pruned specialist recovered to 92.6% exact match, compared with 51.9% after depth and width pruning and 96.3% for its teacher. section 13.4.2













