liveBook · Manning

Chapter 6: The DeepSeek training pipeline: Building a foundation model

What you will learn in chapter 6 of Build a DeepSeek Model (From Scratch)

The chapter introduction previews assembling earlier DeepSeek components into a trainable mini-model and later examining DualPipe for large-scale training.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The chapter introduction previews assembling earlier DeepSeek components into a trainable mini-model and later examining DualPipe for large-scale training.

6.1 The data foundation: Preparing the TinyStories dataset

This section frames data preparation as the prerequisite for assembling and training the model.

Key terms: training data, dataset, tokenization, efficient loading

6.1.1 Choosing the right tools: The TinyStories dataset and TikToken

Explains why TinyStories and the GPT-2 tokenizer are practical choices for training a small English-language model on a consumer GPU.

Key terms: TinyStories, single consumer GPU, tiktoken, gpt2 tokenizer, Byte-Pair Encoding (BPE), vocabulary

6.1.2 Setting up the environment

This section explains how to set up an isolated Python environment and install the libraries needed for the chapter’s TinyStories training project.

Key terms: virtual environment, venv, requirements.txt, PyTorch, NumPy, datasets

6.1.3 The prepare.py script: A step-by-step walkthrough

Walks through prepare.py as it turns TinyStories text into separate, tokenized training and validation files that the training script can read efficiently.

Key terms: VAL_RATIO, train/validation splits, tiktoken, gpt2, vocab_size, encode_corpus, integer IDs, End-of-Text, <|endoftext|>, eot_token

6.2 Assembling the Mini-DeepSeek model

The section introduces model.py as the place to assemble Mini-DeepSeek and explains why its architecture settings are grouped in a ModelArgs dataclass.

Key terms: ModelArgs, dataclass, hyperparameters, configuration object

6.2.1 Building the components: RoPE, MLA, MoE, and MTP

This section builds the RoPE, decoupled MLA attention, MoE, and MTP components, then connects attention and MoE in a pre-norm TransformerBlock.

Key terms: DeepSeekAttention, Content Path, Position Path, RoPE, past_kv, position_offset, DeepSeekMoE, shared_experts, routed_experts, gate

6.2.2 The final architecture: MiniDeepSeek

This section assembles MiniDeepSeek’s top-level model and explains how its forward method switches between cached inference and multi-token-prediction training.

Key terms: MiniDeepSeek, embed, TransformerBlocks, norm_f, lm_head, MTPModules, targets is None, past_kv_cache, position_offset, RoPE

6.3 The training pipeline: Bringing the model to life

This section introduces train.py as the script that connects the prepared dataset and MiniDeepSeek model to next-token training.

Key terms: train.py, prepare.py, model.py, MiniDeepSeek, optimization loop, next token

6.3.1 Configuration and system setup

The opening of train.py establishes the training settings and uses automatic mixed precision to make training faster and less memory-intensive while retaining FP32 where precision matters.

Key terms: autocast, automatic mixed precision (AMP), bfloat16, float16, FP32 master weights, train.py, batch_size, max_iters, AdamW

6.3.2 Loading data and scheduling the learning rate

This section explains the batch-loading and learning-rate scheduling helpers used before the main training loop.

Key terms: get_batch, memory-mapped .bin file, train, val, pin_memory(), get_lr, warmup_iters, cosine decay, min_lr, lr_decay_iters

6.3.3 The main training script

This section explains how train.py initializes MiniDeepSeek, trains and evaluates it, and saves checkpoints.

Key terms: vocab_size, meta.json, ModelArgs, MiniDeepSeek, get_lr, autocast, targets (Y), main loss, MTP loss, loss.backward()

6.4 The engine of scale: Understanding DualPipe Parallelism

Introduces parallel computing as a way to speed up large-model training and identifies data and model parallelism as its two broad approaches.

Key terms: parallel computing, Data Parallelism, Model Parallelism

6.4.1 The standard training loop and its memory limit

Explains the standard weight-update loop, why GPU memory limits batch size, and how accumulating gradients across smaller chunks can produce a large-batch update.

Key terms: mini-batch, forward pass, loss.backward(), optimizer.step(), optimizer.zero_grad(), VRAM, batch size, out of memory, stable gradient, gradient accumulation

6.4.2 Gradient accumulation in code

Shows how gradient accumulation produces a larger effective batch on memory-limited hardware and explains its time-for-memory trade-off.

Key terms: Effective Batch Size, Physical Batch Size, Accumulation Steps, loss.backward(), optimizer.step(), optimizer.zero_grad(), gradient accumulation, effective batch sizes, memory-constrained hardware, forward passes

6.4.3 Data parallelism: The "more GPUs, less time" approach

This section explains how data parallelism speeds up training by processing different parts of a batch on GPUs that hold identical model copies and synchronize their gradients.

Key terms: gradient accumulation, Data Parallelism, mini-batches, model replicas, forward and backward pass, gradients, collective synchronization

6.4.4 Method 1: nn.DataParallel (DP)

Explains how nn.DataParallel runs a training step across GPUs and why its centralized design scales poorly.

Key terms: DataParallel, DP, main GPU, scatter, gather, forward pass, bottleneck, load imbalance, poor scalability

6.4.5 Method 2: nn.DistributedDataParallel (DDP)

Explains how DDP trains independent model replicas on separate GPUs and synchronizes their gradients without a central GPU bottleneck.

Key terms: DistributedDataParallel, DDP, model replica, sharded, process, All-Reduce, Ring-AllReduce, averaged gradient, backward pass, optimizer step

6.4.6 Practical implementation of DDP

Explains the practical setup for running PyTorch DistributedDataParallel (DDP) training across GPUs.

Key terms: torchrun, rank, process group, LOCAL_RANK, DistributedDataParallel, DDP wrapper, All-Reduce, gradient synchronization, DistributedSampler, ddp_model

6.4.7 DDP + gradient accumulation

This section combines DDP and gradient accumulation to overcome training-time and per-GPU memory limits, then works through the batch-size calculation.

Key terms: Distributed Data Parallelism, Gradient Accumulation, time bottleneck, memory bottleneck, effective batch size, physical batch, global batch size, accumulation steps, synchronized update

6.5 Model parallelism

Explains why data parallelism cannot train a model that exceeds one GPU’s memory and introduces model parallelism as the alternative.

Key terms: Data Parallelism, DDP, VRAM, optimizer states, Model Parallelism

6.5.1 Key terminology for model parallelism

Defines the terms used to describe model-parallel pipelines and judge their efficiency.

Key terms: Communication, All-Reduce, Stage, activations, Micro-batch, Pipeline Parallelism, Bubble, Pipeline Bubble, forward pass, backward pass

6.5.2 Pipeline parallelism: The assembly line approach

Explains pipeline parallelism as dividing a model between Transformer layers so data passes through stages on different GPUs.

Key terms: Pipeline Parallelism, Tensor Parallelism, stages, Transformer blocks

6.5.3 The naive schedule (GPipe) and the pipeline bubble

Explains how GPipe fills and drains a micro-batch pipeline and why the resulting idle time is called a pipeline bubble.

Key terms: micro-batches, fill phase, activations, backward pass, gradients, drain phase, GPipe, pipeline bubble, pipeline stages

6.5.4 The 1F1B schedule: Eliminating the bubble

This section explains how 1F1B interleaves pipeline work and why its lower activation-memory use improves practical efficiency even though its theoretical bubble overhead matches GPipe’s.

Key terms: 1F1B, warm-up, steady state, cool-down, micro-batch, pipeline efficiency, bubble steps, GPipe, intermediate activations, peak memory

6.5.5 The DeepSeek innovation: DualPipe Parallelism

Explains why MoE communication remains a bottleneck under 1F1B and introduces the folded stage assignment that enables DualPipe scheduling.

Key terms: 1F1B, Mixture-of-Experts (MoE), token shuffling, MoE communication, pipeline folding, stages, ranks, DualPipe

6.5.6 The DualPipe schedule: Hiding MoE communication

Explains how DualPipe overlaps computation and MoE communication across micro-batches to hide the cost of All-to-All communication.

Key terms: DualPipe, fine-grained computation-communication overlap, micro-batch, All-to-All communication, Attention, MLP (Expert)

6.5.7 A more practical implementation: The DualPipeV schedule

This section explains how DualPipeV folds an N-stage pipeline into a V-shaped path on N/2 GPUs, reducing pipeline idle time while increasing per-GPU parameter memory.

Key terms: DualPipeV, folded pipeline, downstroke, turnaround, upstroke, single, continuous stream, pipeline bubble, interleaving, full bidirectional schedule, N/2 physical GPUs

6.5.8 The trade-offs

This section compares how training-parallelism techniques address different bottlenecks and what each costs.

Key terms: Gradient Accumulation, Data Parallelism (DDP), Pipeline Parallelism (GPipe), pipeline bubble, 1F1B Schedule, activation-memory cost, All-to-All communication, DualPipe, DualPipeV

6.6 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 5: Multi-token prediction and FP8 quantization Chapter 7: Reinforcement learning: From policy gradients to GRPO →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.