liveBook · Manning

Post-training

As explained in Build a DeepSeek Model (From Scratch)

Post-training is the stage of adapting a pretrained model through additional optimization, data generation, or alignment procedures.

Post-training adapts a pretrained model toward useful reasoning and behavior through additional optimization, generated data, supervised examples, and alignment procedures. section 7.0 section 1.2.3

A practical post-training pipeline can proceed through five increasingly capable stages:

  • Start with a lightly fine-tuned base model and a relatively small cold-start dataset. section 1.2.3
  • Apply pure reinforcement learning so the model explores reasoning patterns through trial and error. section 1.2.3 section 7.0
  • Use rejection sampling to select strong responses and create synthetic training data. section 1.2.3
  • Blend synthetic data with supervised examples to balance response quality and domain breadth. section 1.2.3
  • Finish with reinforcement learning across diverse prompts to improve robustness and generalization. section 1.2.3

DeepSeek-R1

DeepSeek-R1 illustrates the pipeline: DeepSeek-V3 served as the base, reinforcement learning encouraged reasoning, self-generated responses supplied training data, and a final RL phase improved performance across task types. Knowledge distillation then transferred capabilities from a large teacher into smaller student checkpoints, including 1.5B, 7B, 8B, 14B, 32B, and 70B models. section 1.2.3

Figure 1.6 The multi-step post-training pipeline used to create DeepSeek-R1 from the DeepSeek-V3 base model. This process involves a combination of reinforcement learning (Pure RL), data generation (Rejection sampling), and fine-tuning to instill advanced reasoning capabilities.

Use the pipeline diagram to trace how fine-tuning, pure reinforcement learning, rejection sampling, data blending, and final RL combine into a reasoning-oriented model. section 1.2.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains a five-stage pipeline and distillation, but stops short of proprietary data, exact weights, and massive distributed infrastructure.
Examples
Uses DeepSeek-R1, DeepSeek-V3, verifiable rewards, synthetic responses, and distilled checkpoints ranging from 1.5B to 70B parameters.
What it assumes
Assumes a pretrained base model while developing post-training concepts from first principles with minimal prior knowledge.
What you can do after
Build it from scratch by implementing reinforcement-learning and distillation techniques for adapting a pretrained language model.
How it connects
Connects supervised fine-tuning, reinforcement learning, rejection sampling, synthetic data, Group Relative Policy Optimization, and knowledge distillation.

Also explained in

Compare how 5 books explain post-training →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.