Knowledge recovery restores capabilities lost when a model is structurally modified. It follows structural optimization, such as pruning, and can precede optional domain specialization. The objective is a smaller or faster model that retains the capabilities needed for its intended use. section 1.2 section 2.0
Recovery begins with pruning decisions: what is removed affects both the initial capability loss and how quickly the model can recover. Depth and width pruning can both degrade knowledge; more aggressive pruning generally increases the loss. section 6.0
Knowledge distillation trains the pruned student to match the original teacher’s prediction distribution, rather than only its final answer. A general-purpose dataset can support broad capability recovery before domain-specific fine-tuning. section 2.5 section 1.2.5
A practical recovery cycle is:
- Measure the pruned model against the original using the same prompts and benchmarks; record size and inference-time changes as well as capability loss. section 2.4
- Load the original teacher and create a pruned student from a copy. Prepare and tokenize recovery data, including attention masks, and batch it for training. section 2.5
- Run teacher inference without gradients and student inference on each batch. Use KL divergence between their output distributions to update student weights. section 2.5
- Reevaluate the student against the original baseline and the pruned model. Check whether the recovered capabilities justify the training cost while retaining the structural savings. section 2.4 section 2.5
Figure 2.3 In the knowledge distillation cycle, the Teacher model generates its reasoning (logits) for a given text. The Pruned model (our student) does the same. The KL Divergence loss function measures the difference between both "reasonings," and this error is used to adjust the student's weights, making it think more like the Teacher over time.
The distillation cycle compares teacher and student outputs and uses their divergence to update the student. section 2.5
Recovering a pruned Gemma model
An 18-layer gemma-3-270m teacher supplies targets to a 16-layer student trained on 15,000 SlimPajama rows. Before recovery, pruning reduced parameters by 4.16% and improved measured inference time, but several benchmark scores fell. Recovery training requires substantially more compute than the pruning operation and should run on a GPU. section 2.5 section 2.4
Output alignment is one recovery strategy. A more advanced approach also uses hard labels and aligns intermediate blocks, introducing the problem of matching blocks when teacher and student depths differ. section 6.0
