liveBook · Manning

Rejection sampling

As explained in Build a DeepSeek Model (From Scratch)

Rejection sampling is a data curation procedure that generates multiple candidates and retains only those meeting specified correctness and quality criteria.

Rejection sampling is a data-curation procedure: generate multiple responses for each prompt, verify them, and retain only candidates that meet correctness and quality criteria. The accepted responses become supervised training data rather than model weights copied from the generator. section 7.7.3 section 8.5.2

A practical rejection-sampling pipeline proceeds as follows:

  • Generate several candidate responses from a capable model, often one strengthened through reasoning-oriented reinforcement learning. section 7.7.1
  • Check correctness with ground-truth comparisons, rule-based verifiers, code execution, or model-based judgments when no single answer exists. section 8.5.2
  • Filter technically correct candidates for readability, language consistency, coherent formatting, and instruction-following quality. section 7.7.1 section 8.5.2
  • Mix accepted reasoning traces with broader supervised examples before fine-tuning, restoring capabilities that verifier-based RL does not directly optimize. section 7.7.1

Math and code curation

For a math prompt, compare each candidate's final answer with the ground truth. For a programming prompt, execute the candidate and test its output. Reject responses that pass correctness checks but contain mixed languages, excessive code blocks, or incoherent formatting; retain only clean, correct traces for training. section 8.5.2

Figure 8.16 The rejection sampling process for curating DeepSeek-R1’s distillation dataset. Multiple responses are generated per prompt, verified for correctness, filtered for quality, and only clean, correct answers are kept.

Use the pipeline diagram to trace candidate generation, correctness verification, quality filtering, and retention of clean responses for distillation. section 8.5.2

Rejection sampling is not reinforcement learning itself. RL improves the generator; rejection sampling harvests its strongest outputs and converts them into a filtered supervised dataset. DeepSeek-R1 used this approach for an approximately 800K-sample distillation dataset. section 7.7.3 section 8.9

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains generation, verification, quality filtering, and dataset integration, without detailing a software implementation.
Examples
Uses ground-truth math checks, executable code tests, model judgments, formatting filters, and an 800K-sample distillation dataset.
What it assumes
Assumes a capable response generator and suitable correctness or quality evaluators, while defining the curation procedure itself.
What you can do after
Build it from scratch by generating candidates, applying correctness and quality filters, and assembling accepted responses into supervised data.
How it connects
Connects reinforcement learning, supervised fine-tuning, knowledge distillation, verifiers, reward models, and synthetic data.

Also explained in

Compare how 5 books explain rejection sampling →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.