liveBook · Manning

Pipeline

As explained in Build a DeepSeek Model (From Scratch)

A training pipeline is an organized sequence of data preparation, model computation, and optimization operations for training a machine-learning model.

A training pipeline organizes the path from raw data to a working language model: prepare and tokenize data, define the model, optimize its parameters, and generate text for evaluation. The implementation-oriented view connects architectural components with the scripts and infrastructure required to train them as one functional system. section 6.0 section 6.6

Figure 1.9 The four-stage roadmap for building a mini-DeepSeek model in this book. We will progress from foundational concepts (Stage 1) and core architecture (Stage 2) to advanced training (Stage 3) and post-training techniques (Stage 4), implementing each key innovation along the way.

Use the roadmap to locate the pipeline between architectural construction and post-training: the training stage combines model innovations with optimization techniques before later refinement. section 1.3

A practical pipeline typically proceeds through these operations:

  • Prepare a dataset by selecting raw text, tokenizing it into numerical representations, and storing it for efficient loading. section 6.1
  • Assemble the model architecture and training scaffolding, then run a training script that optimizes its parameters. section 6.0
  • Generate text with a sampling script after training to inspect the model's behavior and results. section 6.0
  • Scale execution with gradient accumulation, data parallelism, or pipeline parallelism when memory, model size, or training time becomes limiting. section 6.6

Mini-DeepSeek training system

A runnable implementation combines a prepared dataset with a MiniDeepSeek model, a training script, and a text-generation script. Gradient accumulation simulates larger batches on constrained hardware; data parallelism replicates a model across GPUs; pipeline parallelism distributes layers when the model no longer fits on one device. For large mixture-of-experts systems, DualPipe overlaps computation with communication to reduce pipeline inefficiency. section 6.0 section 6.6

The same pattern extends beyond foundation-model pretraining. Reinforcement-learning systems combine data curation, supervised learning, reward-driven optimization, rejection sampling, and hybrid rewards; distillation systems connect teacher and student models through a loss and training loop. section 7.7.4 section 8.6

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Builds the pipeline from dataset preparation through scripts and parallel execution, while deferring production-scale infrastructure details.
Examples
Uses TinyStories preparation, MiniDeepSeek training and sampling, DualPipe scaling, reinforcement-learning workflows, and PyTorch distillation loops.
What it assumes
Assumes familiarity with model components, while teaching dataset preparation, training scaffolding, optimization, sampling, and scaling incrementally.
What you can do after
Build it from scratch as a data-to-generation training system and extend it with optimization and distributed-execution techniques.
How it connects
Connects tokenization, model architecture, optimization, gradient accumulation, data parallelism, pipeline parallelism, reinforcement learning, and knowledge distillation.

Also explained in

Compare how 63 books explain pipeline →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.