liveBook · Manning

Self-distillation fine-tuning

As explained in LLM Customization and Fine-Tuning

A fine-tuning method in which a model conditioned on an expert demonstration trains the same model conditioned on the prompt alone.

Self-distillation fine-tuning (SDFT) trains a model conditioned on an expert demonstration—the teacher—to train the same model conditioned on the prompt alone—the student. It uses the model’s in-context learning as its training signal. section 8.6.2

Consider SDFT when standard supervised fine-tuning risks erasing prior skills, especially in cold-start training before reinforcement learning or when adding capabilities across sequential tasks. It may preserve earlier capabilities, but costs more and remains too new for prescriptive recommendations. section 8.6.2 section 9.7.2

Weigh the reported benefits against these constraints:

  • In reasoning experiments, SDFT preserved reasoning depth better than standard SFT: Olmo-3-7B-Think scored 43.7% accuracy versus 23.5%. section 8.6.2
  • Forgetting benchmarks favored SDFT: prior-capability averages fell by roughly 1 point, compared with roughly 12 points for standard SFT. The cost is 2.5× FLOPs and about 4× wall-clock time; it requires strong in-context learning, typically in models with 7B+ parameters. section 8.6.2 section 6.8.1

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines teacher-to-student conditioning and reports comparative results, but provides no implementation procedure.
Examples
Compares reasoning accuracy on Olmo-3-7B-Think and prior-capability loss against standard SFT.
What it assumes
Requires strong in-context learning capability, typically in models with at least 7B parameters.
What you can do after
Evaluate whether SDFT offers a more capable starting checkpoint despite higher compute and training time.
How it connects
Connects in-context learning with supervised fine-tuning, catastrophic forgetting, DPO, and GRPO.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.