liveBook · Manning

Fine-tuning

As explained in Build a DeepSeek Model (From Scratch)

Fine-tuning is continued training of a pretrained model on a targeted dataset to adapt its behavior or capabilities.

Fine-tuning continues training a pretrained model on targeted data to adapt its behavior. Supervised fine-tuning (SFT) teaches the model to imitate examples, making it useful for shaping response format, style, and known solution patterns. section 7.7.2 section 7.7.6

SFT and reinforcement learning (RL) address different problems: SFT imitates known examples, while RL optimizes behavior against a reward. Combining them can give RL a better starting point without requiring it to discover both readable formatting and correct solutions through exploration. section 7.7.2

A staged reasoning pipeline

DeepSeek-R1 starts with a small cold-start set of readable reasoning examples and fine-tunes a base model before reasoning-oriented RL. It then samples and filters outputs for correctness and readability, mixes reasoning data with broader instruction data, and performs another SFT stage before final RL. The mixed data helps restore general assistant behavior that verifier-based RL does not directly optimize. section 7.7.1 section 7.7.2

Figure 7.14 DeepSeek-R1 training pipeline, adapted from Guo et al., DeepSeek-R1, Figure 2. The flow from R1-Zero into R1 represents filtered training data, not direct parameter transfer.

The pipeline distinguishes the two SFT stages from RL and shows how filtered training data flows into R1. section 7.7.1

SFT can also distill a teacher’s behavior into smaller students. The teacher generates reasoning traces offline; after the dataset is curated, a student trains on it without loading the teacher. Unlike classical distillation through soft targets, this approach transfers generated text rather than the teacher’s internal representations. section 7.7.6 section 8.5.3

Distillation versus direct RL

On the same Qwen2.5-32B base, a student fine-tuned for two to three epochs on teacher-generated reasoning samples reached 72.6% on AIME 2024, versus 47.0% for the direct-RL variant. This comparison demonstrates the value of the dataset in that setup, not a universal advantage of SFT over RL: distillation transfers existing teacher behavior, while RL can help discover or strengthen behavior when rewards are reliable. section 8.7.2 section 7.7.6

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains dataset preparation and placement in training pipelines, but does not detail an SFT implementation.
Examples
Compares SFT on teacher-generated traces with direct RL on the same Qwen2.5-32B architecture.
What you can do after
Evaluate supervised fine-tuning’s role and trade-offs in staged post-training and teacher–student distillation.
How it connects
Connects supervised learning with reinforcement learning, rejection sampling, and knowledge distillation.

Also explained in

Compare how 52 books explain fine-tuning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.