A dataset organizes data examples for model learning. A practical dataset pipeline selects source data, tokenizes raw text into numerical form, and stores it for efficient loading during training. section 6.1
Effective datasets connect data preparation with quality control:
- Tokenization converts raw text into the numerical representation consumed by a model. section 6.1
- Rejection sampling generates multiple responses per prompt, verifies correctness, removes poor formatting, and retains clean examples. section 8.5.2
- A curated distillation dataset supports standard supervised fine-tuning without simultaneous access to the teacher model. section 8.5.3
An 800K distillation dataset
DeepSeek’s distillation dataset contains 800K curated responses. Correctness checks use ground truth, code execution, or model-based judging; formatting checks remove chaotic responses before training. section 8.5.2 section 8.5.5
Listing 8.3 Rejection sampling for reasoning data curation.
def rejection_sample(teacher, prompts,
[CA] n_samples=4): #A
dataset = []
for prompt in prompts:
candidates = []
for _ in range(n_samples): #B
response = teacher.generate(
[CA] prompt, temperature=0.6,
[CA] top_p=0.95,
[CA] max_tokens=32768)
candidates.append(response)
for resp in candidates:
answer = extract_answer(resp)
truth = get_ground_truth(
[CA] prompt) #C
if verify(answer, truth):
if is_clean(resp): #D
dataset.append(
[CA] (prompt, resp))
break
return dataset #EThe rejection-sampling listing makes dataset curation operational: generate candidates, verify answers, filter formatting, and append accepted prompt-response pairs. section 8.5.5





























