liveBook · Manning

Knowledge recovery

As explained in Rearchitecting LLMs

Knowledge recovery is a model-rearchitecting phase that restores or preserves model capabilities after structural changes.

Knowledge recovery restores capabilities lost when a model is structurally modified. It follows structural optimization, such as pruning, and can precede optional domain specialization. The objective is a smaller or faster model that retains the capabilities needed for its intended use. section 1.2 section 2.0

Recovery begins with pruning decisions: what is removed affects both the initial capability loss and how quickly the model can recover. Depth and width pruning can both degrade knowledge; more aggressive pruning generally increases the loss. section 6.0

Knowledge distillation trains the pruned student to match the original teacher’s prediction distribution, rather than only its final answer. A general-purpose dataset can support broad capability recovery before domain-specific fine-tuning. section 2.5 section 1.2.5

A practical recovery cycle is:

  • Measure the pruned model against the original using the same prompts and benchmarks; record size and inference-time changes as well as capability loss. section 2.4
  • Load the original teacher and create a pruned student from a copy. Prepare and tokenize recovery data, including attention masks, and batch it for training. section 2.5
  • Run teacher inference without gradients and student inference on each batch. Use KL divergence between their output distributions to update student weights. section 2.5
  • Reevaluate the student against the original baseline and the pruned model. Check whether the recovered capabilities justify the training cost while retaining the structural savings. section 2.4 section 2.5
Figure 2.3 In the knowledge distillation cycle, the Teacher model generates its reasoning (logits) for a given text. The Pruned model (our student) does the same. The KL Divergence loss function measures the difference between both "reasonings," and this error is used to adjust the student's weights, making it think more like the Teacher over time.

The distillation cycle compares teacher and student outputs and uses their divergence to update the student. section 2.5

Recovering a pruned Gemma model

An 18-layer gemma-3-270m teacher supplies targets to a 16-layer student trained on 15,000 SlimPajama rows. Before recovery, pruning reduced parameters by 4.16% and improved measured inference time, but several benchmark scores fell. Recovery training requires substantially more compute than the pruning operation and should run on a GPU. section 2.5 section 2.4

Output alignment is one recovery strategy. A more advanced approach also uses hard labels and aligns intermediate blocks, introducing the problem of matching blocks when teacher and student depths differ. section 6.0

How Rearchitecting LLMs covers it

How deep it goes
Walks through pruning decisions, data preparation, and a distillation loop, and introduces intermediate-block alignment as a more complex strategy.
Examples
Uses an 18-layer Gemma teacher, a 16-layer student, and 15,000 SlimPajama rows in a recovery experiment.
What you can do after
Apply it via OptiPFair and a teacher-student distillation loop to train a pruned model.
How it connects
Connects structural pruning, knowledge distillation, KL divergence, and fine-tuning.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.