Self-distillation fine-tuning (SDFT) trains a model conditioned on an expert demonstration—the teacher—to train the same model conditioned on the prompt alone—the student. It uses the model’s in-context learning as its training signal. section 8.6.2
Consider SDFT when standard supervised fine-tuning risks erasing prior skills, especially in cold-start training before reinforcement learning or when adding capabilities across sequential tasks. It may preserve earlier capabilities, but costs more and remains too new for prescriptive recommendations. section 8.6.2 section 9.7.2
Weigh the reported benefits against these constraints:
- In reasoning experiments, SDFT preserved reasoning depth better than standard SFT: Olmo-3-7B-Think scored 43.7% accuracy versus 23.5%. section 8.6.2
- Forgetting benchmarks favored SDFT: prior-capability averages fell by roughly 1 point, compared with roughly 12 points for standard SFT. The cost is 2.5× FLOPs and about 4× wall-clock time; it requires strong in-context learning, typically in models with 7B+ parameters. section 8.6.2 section 6.8.1
