liveBook · Manning

DeepSeek distillation dataset

As explained in Build a DeepSeek Model (From Scratch)

The DeepSeek distillation dataset is a book-specific collection of approximately 800,000 curated reasoning and non-reasoning training samples.

The DeepSeek distillation dataset contains approximately 800,000 teacher-generated samples: about 600,000 reasoning traces and 200,000 non-reasoning examples. It supports supervised fine-tuning of dense student models rather than classical temperature-scaled KL distillation. section 8.5.2 section 8.9

Build the dataset through rejection sampling:

  • Generate multiple candidate responses for each prompt with the DeepSeek-R1 teacher. section 8.5.2
  • Verify mathematical answers against ground truth and execute code responses to test their outputs. section 8.5.2
  • Use DeepSeek-V3 as a generative reward model for STEM reasoning and logical puzzles without one definitive answer. section 8.5.2
  • Discard responses that are correct but chaotic, mixed-language, excessively code-heavy, or incoherently formatted. section 8.5.2

Estimated composition and results

The estimated domain split is 395,000 math samples, 211,000 code, 10,000 STEM, 10,000 logic, and 178,000 general samples. A distilled 32B model reaches 72.6% on AIME, compared with 47.0% for a reinforcement-learning-only counterpart; a 1.5B student reaches 83.9% on MATH but 16.9% on LiveCodeBench. section 8.9

Figure 8.16 The rejection sampling process for curating DeepSeek-R1’s distillation dataset. Multiple responses are generated per prompt, verified for correctness, filtered for quality, and only clean, correct answers are kept.

Trace the curation pipeline from multiple teacher responses through correctness verification and formatting filters to retained samples. section 8.5.2

Figure 8.17 DeepSeek-R1’s complete training pipeline. Distillation (Stage 6) is the final stage, where open-source dense models are fine-tuned on 800K samples curated from the teacher via rejection sampling. Data generation costs only $10K versus $202K for RL training. (Cost figures are estimates derived from the H800-hour counts disclosed in the DeepSeek-R1 paper at published cloud GPU rates; exact internal costs may differ.)

Locate distillation as Stage 6, where open-source dense models are fine-tuned on the curated dataset. section 8.5.2

Figure 8.18 The composition of DeepSeek-R1’s 800K distillation dataset by domain. Math dominates at 395K samples (49%), followed by Code at 211K (26%), with General, STEM, and Logic making up the remainder. (Estimated split; the R1 paper itself reports only the ~600K reasoning / ~200K non-reasoning totals.)

Inspect the estimated distribution across math, code, STEM, logic, and general domains. section 8.5.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains generation, verification, reward-model judging, and formatting filters, but does not detail the fine-tuning implementation.
Examples
Uses verified math and code solutions, open-ended STEM and logic judgments, and estimated domain counts as concrete examples.
What it assumes
Assumes classical distillation concepts, while contrasting them with supervised training on teacher-generated reasoning traces.
What you can do after
Evaluate how rejection-sampled teacher traces transfer reasoning unevenly across student sizes and domains.
How it connects
Connects rejection sampling with supervised fine-tuning, chain-of-thought reasoning, generative reward models, and reinforcement learning.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.