liveBook · Manning

Continued pretraining

As explained in Rearchitecting LLMs

Continued pretraining is additional unsupervised training of a pretrained model on text or other pretraining data.

Continued pretraining (CPT) adds further unsupervised training to a pretrained model, often as a knowledge-recovery step after structural changes such as aggressive depth pruning. section 4.4

Use CPT when recovery requirements exceed lightweight adaptation:

  • Prefer LoRA for moderate pruning, but favor CPT when pruning exceeds 40% and the model needs substantial performance recovery. section 4.4
  • Treat CPT as similar to full training, with higher recovery potential but greater training cost than parameter-efficient adaptation. section 4.4
  • Distinguish general capability recovery from domain specialization; knowledge distillation is used for the former, while LoRA is reserved for the latter. section 4.4

Post-pruning recovery

After removing a large fraction of transformer blocks, continued pretraining can retrain the modified model to recover lost performance, particularly beyond 40% pruning. section 4.4

How Rearchitecting LLMs covers it

How deep it goes
Compares CPT with LoRA and defers detailed knowledge-distillation recovery procedures.
Examples
Uses aggressive pruning above 40% as the threshold where CPT outperforms LoRA for recovery.
What it assumes
Assumes a pretrained model and understanding of structural pruning, while contrasting recovery with parameter-efficient adaptation.
What you can do after
Evaluate whether full-training-like recovery is warranted after moderate or aggressive model pruning.
How it connects
Connects continued pretraining with depth pruning, LoRA, knowledge distillation, and model recovery.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.