liveBook · Manning

Recovery

As explained in Rearchitecting LLMs

A recovery process is a post-restructuring procedure that restores or improves a model’s performance after substantial structural components have been removed.

Recovery restores or improves a model’s performance after structural pruning. Pruning removes capacity and can increase perplexity; fine-tuning or knowledge distillation can then readjust the remaining weights. The practical pipeline is pruning plus recovery, with pruning designed to minimize damage on the target domain. section 5.2.5 section 8.4.2

A practical distillation workflow separates candidate selection from recovery validation:

  • Compare pruning candidates before recovery using perplexity and capability benchmarks. A less-degraded starting model may produce the strongest final result, even if another candidate gains more during recovery. section 6.1.3
  • Prepare a pruned student from the teacher and enable gradients on the student’s parameters. Run a small distillation test before committing to a candidate. section 6.1.3
  • Train the student against both ground-truth labels and the teacher’s output distribution. In one OptiPFair configuration, alpha=0.6 weights the label loss and beta=0.40 weights logit-based distillation. section 8.4.2
  • Evaluate both benchmarks and generated text. For attention-pruned generation models, test for repetition loops and factual consistency over sustained output; benchmark improvements alone may not establish coherent generation. section 8.5.2 section 8.6
trained_student, stats = opf.distill_model(
    student_model=student_to_train,
    teacher_model=teacher_model,
    dataloader=train_dataloader,
    alpha=0.6,
    beta=0.40,
)

This call runs distillation with a student, teacher, training dataloader, and explicit loss weights. section 8.4.2

Selecting a recoverable pruning candidate

After four blocks were removed, a data-driven candidate had perplexity 126 and 78.6% capability retention. A test using 2,000 Cosmopedia records for five epochs brought it to perplexity 18.9 and 86.7% retention. Other candidates gained more percentage points but finished with lower retention. Training took 3–10 minutes per model, depending on GPU capacity. section 6.1.3

Recovery is task-dependent. Attention removal without recovery can be viable for classification or ranking, while text generation may require even a brief recovery to restore autoregressive coherence. One six-attention-layer removal example regained coherent generation after distillation on 10,000 Cosmopedia records, taking ten minutes on an A100 GPU; this does not establish that every pruning level is recoverable. section 8.4.2 section 8.5.2

To tune the recovery budget, vary the loss balance, dataset size, or training duration across configurations. Compare their benchmarks and outputs to find the smallest configuration that removes the observed repetition loop without relying on a single metric. section 8.6

How Rearchitecting LLMs covers it

How deep it goes
Walks through candidate selection, distillation settings, timing, and validation, with an exercise varying the recovery budget.
Examples
Compares four block-pruning candidates and tests recovery of an attention-pruned model with a Paris generation prompt.
What you can do after
Apply it via OptiPFair knowledge distillation and evaluate recovered models using benchmarks and direct generation checks.
How it connects
Connects pruning to knowledge distillation, fine-tuning, perplexity, and autoregressive text generation.

Also explained in

Compare how 10 books explain recovery →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.