Continued pretraining (CPT) adds further unsupervised training to a pretrained model, often as a knowledge-recovery step after structural changes such as aggressive depth pruning. section 4.4
Use CPT when recovery requirements exceed lightweight adaptation:
- Prefer LoRA for moderate pruning, but favor CPT when pruning exceeds 40% and the model needs substantial performance recovery. section 4.4
- Treat CPT as similar to full training, with higher recovery potential but greater training cost than parameter-efficient adaptation. section 4.4
- Distinguish general capability recovery from domain specialization; knowledge distillation is used for the former, while LoRA is reserved for the latter. section 4.4
Post-pruning recovery
After removing a large fraction of transformer blocks, continued pretraining can retrain the modified model to recover lost performance, particularly beyond 40% pruning. section 4.4
