Fine-tuning continues training a pretrained model on targeted data to adapt its behavior. Supervised fine-tuning (SFT) teaches the model to imitate examples, making it useful for shaping response format, style, and known solution patterns. section 7.7.2 section 7.7.6
SFT and reinforcement learning (RL) address different problems: SFT imitates known examples, while RL optimizes behavior against a reward. Combining them can give RL a better starting point without requiring it to discover both readable formatting and correct solutions through exploration. section 7.7.2
A staged reasoning pipeline
DeepSeek-R1 starts with a small cold-start set of readable reasoning examples and fine-tunes a base model before reasoning-oriented RL. It then samples and filters outputs for correctness and readability, mixes reasoning data with broader instruction data, and performs another SFT stage before final RL. The mixed data helps restore general assistant behavior that verifier-based RL does not directly optimize. section 7.7.1 section 7.7.2
Figure 7.14 DeepSeek-R1 training pipeline, adapted from Guo et al., DeepSeek-R1, Figure 2. The flow from R1-Zero into R1 represents filtered training data, not direct parameter transfer.

The pipeline distinguishes the two SFT stages from RL and shows how filtered training data flows into R1. section 7.7.1
SFT can also distill a teacher’s behavior into smaller students. The teacher generates reasoning traces offline; after the dataset is curated, a student trains on it without loading the teacher. Unlike classical distillation through soft targets, this approach transfers generated text rather than the teacher’s internal representations. section 7.7.6 section 8.5.3
Distillation versus direct RL
On the same Qwen2.5-32B base, a student fine-tuned for two to three epochs on teacher-generated reasoning samples reached 72.6% on AIME 2024, versus 47.0% for the direct-RL variant. This comparison demonstrates the value of the dataset in that setup, not a universal advantage of SFT over RL: distillation transfers existing teacher behavior, while RL can help discover or strengthen behavior when rewards are reliable. section 8.7.2 section 7.7.6



















































