liveBook · Manning

Dataset

As explained in Build a DeepSeek Model (From Scratch)

A dataset is a collection of data examples organized for analysis or for training, validating, or evaluating machine-learning models.

A dataset organizes data examples for model learning. A practical dataset pipeline selects source data, tokenizes raw text into numerical form, and stores it for efficient loading during training. section 6.1

Effective datasets connect data preparation with quality control:

  • Tokenization converts raw text into the numerical representation consumed by a model. section 6.1
  • Rejection sampling generates multiple responses per prompt, verifies correctness, removes poor formatting, and retains clean examples. section 8.5.2
  • A curated distillation dataset supports standard supervised fine-tuning without simultaneous access to the teacher model. section 8.5.3

An 800K distillation dataset

DeepSeek’s distillation dataset contains 800K curated responses. Correctness checks use ground truth, code execution, or model-based judging; formatting checks remove chaotic responses before training. section 8.5.2 section 8.5.5

Listing 8.3 Rejection sampling for reasoning data curation.
def rejection_sample(teacher, prompts,
     [CA] n_samples=4):  #A
    dataset = []
    for prompt in prompts:
        candidates = []
        for _ in range(n_samples):  #B
            response = teacher.generate(
             [CA] prompt, temperature=0.6,
             [CA] top_p=0.95,
             [CA] max_tokens=32768)
            candidates.append(response)
        for resp in candidates:
            answer = extract_answer(resp)
            truth = get_ground_truth(
             [CA] prompt)  #C
            if verify(answer, truth):
                if is_clean(resp):  #D
                    dataset.append(
                     [CA] (prompt, resp))
                    break
    return dataset  #E

The rejection-sampling listing makes dataset curation operational: generate candidates, verify answers, filter formatting, and append accepted prompt-response pairs. section 8.5.5

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains preparation, tokenization, storage, rejection sampling, verification, and quality filtering, with an executable curation procedure.
Examples
Uses TinyStories training data and an 800K teacher-generated distillation dataset filtered for correctness and readable formatting.
What it assumes
Assumes model-training context but introduces dataset preparation, tokenization, verification, and quality filtering as an end-to-end workflow.
What you can do after
Build it from scratch by selecting data, tokenizing text, storing examples, and curating verified responses for supervised fine-tuning.
How it connects
Connects datasets to tokenization, data loading, rejection sampling, supervised fine-tuning, and knowledge distillation.

Also explained in

Compare how 30 books explain dataset →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.