Chapter 6: The DeepSeek training pipeline: Building a foundation model
The chapter introduction previews assembling earlier DeepSeek components into a trainable mini-model and later examining DualPipe for large-scale training.
After this chapter you will be able to
- Training data preparation stages. Describe the three broad stages of preparing training data: selecting an appropriate dataset, tokenizing its raw text into numerical form, and saving it for efficient loading during training.
- TinyStories dataset suitability. Justify the choice of TinyStories for this project: its simple, coherent English stories offer useful language-learning patterns while keeping training manageable on a single consumer GPU.
- GPT-2 tokenizer choice. Explain why the project uses tiktoken's standard GPT-2 tokenizer rather than the official DeepSeek tokenizer or a newly trained one.
- Project Environment and Dependencies. Describe why the project uses a virtual environment and how its required libraries are installed. State what PyTorch, NumPy, datasets, and tiktoken are needed for.
- Dataset splits and tokenizer setup. Describe how prepare.py loads the dataset, reserves a validation split, initializes the GPT-2 tokenizer, and records its vocabulary size for model initialization.
- Story encoding and boundaries. Trace how encode_corpus turns individual stories into a token stream and explain the role of the End-of-Text token.
- Memory-mapped training files. Explain why prepare.py writes memory-mapped training and validation files and identify the purpose of the accompanying metadata file.
- ModelArgs configuration purpose. Explain why model.py uses a ModelArgs dataclass instead of passing many hyperparameters separately to the model constructor.
- Decoupled attention and cached positions. Explain how DeepSeekAttention combines separate content and position scores, and how its cache keeps RoPE positions correct when generating a new token.
- Shared and routed expert outputs. Trace how DeepSeekMoE forms an output from shared experts and selected routed experts.
- Pre-norm TransformerBlock data flow. Trace the attention and MoE stages of the TransformerBlock, including its residual connections and cache handoff.
- Cached inference path. Trace how the forward method selects inference when targets are absent, sets the new token’s positional offset from the past KV cache, and computes only the last token’s logits.
- MTP training loss path. Trace how providing targets selects full-sequence training, produces the main next-token loss, and adds weighted losses from successive MTP modules.
- Role of train.py. Explain how train.py uses the dataset created by prepare.py and the model defined in model.py to train MiniDeepSeek to predict the next token.
- Automatic mixed precision workflow. Explain how autocast divides training work between 16-bit and FP32 precision, including which weights the optimizer updates.
- Training batch construction. Describe how get_batch loads a mini-batch for a training step and why it uses pin_memory().
- Warmup and cosine learning rate. Explain the scheduler’s warmup, cosine decay, and minimum-rate behavior, including when the min_lr guard is reached.
- Training iteration sequence. Trace one training iteration from the learning-rate update through fetching a batch, computing the combined loss, calculating gradients, and updating weights.
- Periodic validation loss. Explain when and how the script evaluates validation loss, and why it does so separately from training.
- Best and periodic checkpoints. Distinguish the two checkpoint-saving conditions and describe what a checkpoint contains and why those contents are saved.
- Parallel computing for training scale. Explain why large-model training uses parallel computing: a single GPU can take too long, so multiple GPUs work simultaneously on the training problem. Identify data and model parallelism as the two broad categories named here, without needing to explain their mechanics.
- Large-batch GPU memory limit. Explain why a desirable batch size may be impossible to process in one standard training step on a GPU.
- Equivalent accumulated batch gradient. Explain how gradients from smaller physical batches can be combined to give the averaged gradient of a larger effective batch, and why PyTorch can accumulate them.
- Gradient accumulation update cycle. Trace how four physical batches of 16 produce one optimizer update for an effective batch of 64, including when gradients are retained, weights are updated, and gradients are cleared.
- Memory and training time trade-off. Explain why gradient accumulation permits a larger effective batch on memory-constrained hardware and why it slows training relative to processing that batch at once.
- Replicated model gradient synchronization. Trace one data-parallel iteration from splitting a batch across GPUs with identical model copies through local forward and backward passes, gradient averaging, and the same parameter update on every replica.
- Centralized DataParallel Training Step. Trace a DataParallel training step: the main GPU distributes batch slices and model copies, GPUs compute in parallel, and the main GPU gathers results and updates the model.
- Main GPU Scalability Bottleneck. Explain why DataParallel's centralized design causes load imbalance and poor scalability despite being simple to implement.
- Independent replicas and data shards. Describe how DDP gives each GPU its own process, model replica, optimizer, and unique slice of the batch, then has each process compute a local loss and gradients.
- Gradient averaging and synchronized updates. Trace how DDP averages locally computed gradients across peers, overlaps communication with backward computation, and lets every process apply the same update to keep model replicas synchronized.
- DDP process and device setup. Describe how torchrun launches one process per GPU, and how the training script initializes communication and assigns each process to its GPU.
- DDP model and data integration. Explain why the model is wrapped in DistributedDataParallel, why a DistributedSampler is used, and what changes in the training loop.
- Complementary training bottlenecks. Explain how DDP and gradient accumulation address different limits in large-scale training: DDP spreads computation across GPUs, while accumulation achieves an effective batch size too large to fit on one device.
- Effective batch calculation. Apply the section’s example to calculate the global batch and accumulation steps, and describe when the accumulated gradients lead to an update.
- Data parallelism memory limit. Explain why DDP and gradient accumulation do not solve the problem of a model that cannot fit on one GPU.
- Model parallelism core idea. Describe model parallelism as splitting the model itself across multiple GPUs rather than splitting the data.
- Inter-GPU communication overhead. Explain what communication means in parallel computing and why advanced parallelism techniques try to reduce or overlap it with computation.
- Pipeline bubble idle time. Explain what the pipeline bubble is, when GPUs experience it, and why advanced pipeline techniques seek to minimize it.
- Pipeline stages and data flow. Explain how pipeline parallelism partitions consecutive Transformer blocks into stages on different GPUs and passes data from one stage to the next, rather than splitting computation within a layer.
- Forward pipeline fill. Trace how GPipe feeds micro-batches through successive GPU stages and explain why later stages initially sit idle.
- Backward pipeline drain. Explain why GPipe's backward pass starts at the last stage and leaves earlier-stage GPUs waiting.
- Combined pipeline bubble. Explain what the pipeline bubble represents in GPipe and why it worsens as the number of stages grows.
- 1F1B pipeline phases. Trace the warm-up, steady-state, and cool-down phases of 1F1B, explaining when a pipeline stage switches between forward and backward work.
- Fixed-size pipeline efficiency. Explain why 1F1B and GPipe have the same theoretical maximum utilization for a fixed number of micro-batches and stages, despite their different schedules.
- Activation memory advantage. Explain how 1F1B reduces peak activation memory relative to GPipe and why that can improve real-world utilization.
- MoE communication bottleneck. Explain why a balanced 1F1B pipeline can still leave GPUs idle when training a large MoE model.
- Folded pipeline stage assignment. Describe how pipeline folding assigns model stages to GPUs differently from a standard sequential pipeline.
- Fine-grained DualPipe overlap. Explain how DualPipe schedules Attention, Expert computation, and All-to-All communication across micro-batches, and why this helps large MoE training.
- V-shaped micro-batch path. Trace a micro-batch through the folded pipeline and describe how its backward pass follows that path in reverse.
- Single-stream bubble reduction. Explain how DualPipeV simplifies the full bidirectional schedule and why it still reduces the pipeline bubble.
- GPU and memory trade-off. Describe the hardware benefit of the folded pipeline and the parameter-memory cost on each GPU.
- Memory and hardware trade-offs. Explain the bottleneck and trade-off for gradient accumulation, DDP, and GPipe.
- Pipeline scheduling trade-offs. Trace why 1F1B improves on naive pipelining and why MoE training motivates DualPipe or DualPipeV, including their remaining cost.
What it builds on
- The DeepSeek solution: Decoupled rotary position embedding (chapter 3) · The decoupled MLA and RoPE design implemented here.
- The DeepSeek innovations: Towards ultimate expert specialization (chapter 4) · The shared-expert and fine-grained routed-expert design.
- Innovation #3: Auxiliary-loss-free load balancing (chapter 4) · The auxiliary-loss-free load-balancing approach.
- The DeepSeek MTP architecture: A visual and mathematical walkthrough (chapter 5) · The multi-token prediction architecture introduced earlier.
- The new inference loop in action: What happens when a new token arrives (chapter 3) · The cached inference loop and positional handling in the model’s attention components.
- The MTP modules: A sequential chain of prediction (chapter 5) · The sequential chain of multi-token-prediction modules.
- Pillar 1: The mixed precision framework (chapter 5) · Introduces the idea of a mixed-precision training framework.
Section by section
Introduction
The chapter introduction previews assembling earlier DeepSeek components into a trainable mini-model and later examining DualPipe for large-scale training.
6.1 The data foundation: Preparing the TinyStories dataset
This section frames data preparation as the prerequisite for assembling and training the model.
- Training data preparation stages: Describe the three broad stages of preparing training data: selecting an appropriate dataset, tokenizing its raw text into numerical form, and saving it for efficient loading during training.
Why it matters: These stages explain why raw text cannot simply be handed to the training pipeline as-is.
Key terms: training data, dataset, tokenization, efficient loading
6.1.1 Choosing the right tools: The TinyStories dataset and TikToken
Explains why TinyStories and the GPT-2 tokenizer are practical choices for training a small English-language model on a consumer GPU.
- TinyStories dataset suitability: Justify the choice of TinyStories for this project: its simple, coherent English stories offer useful language-learning patterns while keeping training manageable on a single consumer GPU.
Why it matters: The dataset choice makes meaningful training feasible at the project's scale. - GPT-2 tokenizer choice: Explain why the project uses tiktoken's standard GPT-2 tokenizer rather than the official DeepSeek tokenizer or a newly trained one.
Why it matters: The tokenizer choice matches the dataset's scope without adding unnecessary preparation work.
Key terms: TinyStories, single consumer GPU, tiktoken, gpt2 tokenizer, Byte-Pair Encoding (BPE), vocabulary
6.1.2 Setting up the environment
This section explains how to set up an isolated Python environment and install the libraries needed for the chapter’s TinyStories training project.
- Project Environment and Dependencies: Describe why the project uses a virtual environment and how its required libraries are installed. State what PyTorch, NumPy, datasets, and tiktoken are needed for.
Why it matters: The isolated environment and installed libraries let you follow the chapter’s data-preparation and model-building code without disrupting other Python projects.
Key terms: virtual environment, venv, requirements.txt, PyTorch, NumPy, datasets
6.1.3 The prepare.py script: A step-by-step walkthrough
Walks through prepare.py as it turns TinyStories text into separate, tokenized training and validation files that the training script can read efficiently.
- Dataset splits and tokenizer setup: Describe how prepare.py loads the dataset, reserves a validation split, initializes the GPT-2 tokenizer, and records its vocabulary size for model initialization.
Why it matters: These choices make the prepared data usable for both model setup and a meaningful validation check. - Story encoding and boundaries: Trace how encode_corpus turns individual stories into a token stream and explain the role of the End-of-Text token.
Why it matters: The model can train on a continuous token stream without losing track of where stories end. - Compact token ID storage: Justify storing the token IDs as uint16 rather than wider integers using the vocabulary size given in the section.
Why it matters: The chosen integer type reduces the size of the prepared dataset without excluding any tokenizer IDs. - Memory-mapped training files: Explain why prepare.py writes memory-mapped training and validation files and identify the purpose of the accompanying metadata file.
Why it matters: The saved files provide the training script with token data and model metadata without requiring the whole dataset in RAM.
Key terms: VAL_RATIO, train/validation splits, tiktoken, gpt2, vocab_size, encode_corpus, integer IDs, End-of-Text, <|endoftext|>, eot_token
6.2 Assembling the Mini-DeepSeek model
The section introduces model.py as the place to assemble Mini-DeepSeek and explains why its architecture settings are grouped in a ModelArgs dataclass.
- ModelArgs configuration purpose: Explain why model.py uses a ModelArgs dataclass instead of passing many hyperparameters separately to the model constructor.
Why it matters: This explains how the model’s architectural settings remain manageable as its size changes.
Key terms: ModelArgs, dataclass, hyperparameters, configuration object
6.2.1 Building the components: RoPE, MLA, MoE, and MTP
This section builds the RoPE, decoupled MLA attention, MoE, and MTP components, then connects attention and MoE in a pre-norm TransformerBlock.
- Decoupled attention and cached positions: Explain how DeepSeekAttention combines separate content and position scores, and how its cache keeps RoPE positions correct when generating a new token.
Why it matters: This connects the model’s attention calculation to correct one-token-at-a-time inference. - Shared and routed expert outputs: Trace how DeepSeekMoE forms an output from shared experts and selected routed experts.
Why it matters: It explains how the MoE layer combines always-used expertise with selectively routed computation. - Training-only routing bias update: Explain when DeepSeekMoE updates its dynamic routing bias and why that update is kept outside gradient computation.
Why it matters: The distinction prevents the routing adjustment from being mistaken for a gradient-based auxiliary loss. - Causal MTP module inputs: Describe how the simplified MTPModule prepares a hidden state for its next prediction step.
Why it matters: It shows how this implementation carries information forward for an additional causal prediction step. - Pre-norm TransformerBlock data flow: Trace the attention and MoE stages of the TransformerBlock, including its residual connections and cache handoff.
Why it matters: This is where the specialized components become a usable block while preserving cached-inference behavior.
Key terms: DeepSeekAttention, Content Path, Position Path, RoPE, past_kv, position_offset, DeepSeekMoE, shared_experts, routed_experts, gate
6.2.2 The final architecture: MiniDeepSeek
This section assembles MiniDeepSeek’s top-level model and explains how its forward method switches between cached inference and multi-token-prediction training.
- MiniDeepSeek model components: Describe how the MiniDeepSeek class brings together the embedding layer, TransformerBlock stack, final normalization, output layer, and MTP modules.
Why it matters: The container shows how the previously built parts become one usable model. - Cached inference path: Trace how the forward method selects inference when targets are absent, sets the new token’s positional offset from the past KV cache, and computes only the last token’s logits.
Why it matters: This explains how the same model generates efficiently without losing the new token’s correct position. - MTP training loss path: Trace how providing targets selects full-sequence training, produces the main next-token loss, and adds weighted losses from successive MTP modules.
Why it matters: The combined loss explains how MiniDeepSeek trains on both next-token and farther-ahead predictions.
Key terms: MiniDeepSeek, embed, TransformerBlocks, norm_f, lm_head, MTPModules, targets is None, past_kv_cache, position_offset, RoPE
6.3 The training pipeline: Bringing the model to life
This section introduces train.py as the script that connects the prepared dataset and MiniDeepSeek model to next-token training.
- Role of train.py: Explain how train.py uses the dataset created by prepare.py and the model defined in model.py to train MiniDeepSeek to predict the next token.
Why it matters: It connects the data and model you have built to the script that trains them.
Key terms: train.py, prepare.py, model.py, MiniDeepSeek, optimization loop, next token
6.3.1 Configuration and system setup
The opening of train.py establishes the training settings and uses automatic mixed precision to make training faster and less memory-intensive while retaining FP32 where precision matters.
- Automatic mixed precision workflow: Explain how autocast divides training work between 16-bit and FP32 precision, including which weights the optimizer updates.
Why it matters: This explains how the training script gains speed and saves memory without using reduced precision everywhere. - Training script configuration: Describe what the opening of train.py configures before training begins.
Why it matters: These settings establish the conditions under which the later training script runs.
Key terms: autocast, automatic mixed precision (AMP), bfloat16, float16, FP32 master weights, train.py, batch_size, max_iters, AdamW
6.3.2 Loading data and scheduling the learning rate
This section explains the batch-loading and learning-rate scheduling helpers used before the main training loop.
- Training batch construction: Describe how get_batch loads a mini-batch for a training step and why it uses pin_memory().
Why it matters: The training loop needs fresh input and target batches delivered efficiently to the GPU. - Warmup and cosine learning rate: Explain the scheduler’s warmup, cosine decay, and minimum-rate behavior, including when the min_lr guard is reached.
Why it matters: The schedule supports stable early training and progressively smaller updates without mistaking a defensive branch for a normal training phase.
Key terms: get_batch, memory-mapped .bin file, train, val, pin_memory(), get_lr, warmup_iters, cosine decay, min_lr, lr_decay_iters
6.3.3 The main training script
This section explains how train.py initializes MiniDeepSeek, trains and evaluates it, and saves checkpoints.
- Model and optimizer setup: Describe how the script loads the vocabulary size, configures and instantiates MiniDeepSeek, moves it to the chosen device, and sets up the optimizer.
Why it matters: These steps connect the prepared data and model configuration to the training run. - Training iteration sequence: Trace one training iteration from the learning-rate update through fetching a batch, computing the combined loss, calculating gradients, and updating weights.
Why it matters: This sequence is how the assembled model learns from the TinyStories training batches. - Periodic validation loss: Explain when and how the script evaluates validation loss, and why it does so separately from training.
Why it matters: Validation loss shows how the model performs on held-out data and determines when a new best checkpoint is saved. - Best and periodic checkpoints: Distinguish the two checkpoint-saving conditions and describe what a checkpoint contains and why those contents are saved.
Why it matters: Checkpoints preserve both the best evaluated model and a recent recovery point if training is interrupted.
Key terms: vocab_size, meta.json, ModelArgs, MiniDeepSeek, get_lr, autocast, targets (Y), main loss, MTP loss, loss.backward()
6.4 The engine of scale: Understanding DualPipe Parallelism
Introduces parallel computing as a way to speed up large-model training and identifies data and model parallelism as its two broad approaches.
- Parallel computing for training scale: Explain why large-model training uses parallel computing: a single GPU can take too long, so multiple GPUs work simultaneously on the training problem. Identify data and model parallelism as the two broad categories named here, without needing to explain their mechanics.
Why it matters: This is the motivation for moving from the single-GPU MiniDeepSeek pipeline to multi-GPU training.
Key terms: parallel computing, Data Parallelism, Model Parallelism
6.4.1 The standard training loop and its memory limit
Explains the standard weight-update loop, why GPU memory limits batch size, and how accumulating gradients across smaller chunks can produce a large-batch update.
- Standard mini-batch weight update: Trace one training step from a mini-batch through the forward pass, loss, backward pass, optimizer update, and gradient reset.
Why it matters: The ordinary step-and-reset sequence is the baseline that gradient accumulation changes. - Large-batch GPU memory limit: Explain why a desirable batch size may be impossible to process in one standard training step on a GPU.
Why it matters: This memory constraint explains why the section seeks a large-batch gradient without processing the whole batch at once. - Equivalent accumulated batch gradient: Explain how gradients from smaller physical batches can be combined to give the averaged gradient of a larger effective batch, and why PyTorch can accumulate them.
Why it matters: This is the mathematical reason smaller batches can yield a large-batch update without fitting the whole batch in VRAM.
Key terms: mini-batch, forward pass, loss.backward(), optimizer.step(), optimizer.zero_grad(), VRAM, batch size, out of memory, stable gradient, gradient accumulation
6.4.2 Gradient accumulation in code
Shows how gradient accumulation produces a larger effective batch on memory-limited hardware and explains its time-for-memory trade-off.
- Gradient accumulation update cycle: Trace how four physical batches of 16 produce one optimizer update for an effective batch of 64, including when gradients are retained, weights are updated, and gradients are cleared.
Why it matters: The timing of backward passes, optimizer steps, and gradient clearing is what makes a large effective batch possible. - Memory and training time trade-off: Explain why gradient accumulation permits a larger effective batch on memory-constrained hardware and why it slows training relative to processing that batch at once.
Why it matters: It helps distinguish fitting a larger effective batch from making training faster.
Key terms: Effective Batch Size, Physical Batch Size, Accumulation Steps, loss.backward(), optimizer.step(), optimizer.zero_grad(), gradient accumulation, effective batch sizes, memory-constrained hardware, forward passes
6.4.3 Data parallelism: The "more GPUs, less time" approach
This section explains how data parallelism speeds up training by processing different parts of a batch on GPUs that hold identical model copies and synchronize their gradients.
- Data parallelism motivation: Explain why the section introduces data parallelism after gradient accumulation: accumulation addresses memory limits but increases training time, whereas multiple GPUs can process batch chunks simultaneously.
Why it matters: It clarifies why using multiple GPUs addresses a different bottleneck from gradient accumulation. - Replicated model gradient synchronization: Trace one data-parallel iteration from splitting a batch across GPUs with identical model copies through local forward and backward passes, gradient averaging, and the same parameter update on every replica.
Why it matters: The shared gradient update is what lets separate GPUs work on different data without their model copies drifting apart.
Key terms: gradient accumulation, Data Parallelism, mini-batches, model replicas, forward and backward pass, gradients, collective synchronization
6.4.4 Method 1: nn.DataParallel (DP)
Explains how nn.DataParallel runs a training step across GPUs and why its centralized design scales poorly.
- Centralized DataParallel Training Step: Trace a DataParallel training step: the main GPU distributes batch slices and model copies, GPUs compute in parallel, and the main GPU gathers results and updates the model.
Why it matters: Following where data, outputs, and gradients go makes DP's central coordination clear. - Main GPU Scalability Bottleneck: Explain why DataParallel's centralized design causes load imbalance and poor scalability despite being simple to implement.
Why it matters: The bottleneck explains why the simplest multi-GPU method is rarely chosen for large-scale training.
Key terms: DataParallel, DP, main GPU, scatter, gather, forward pass, bottleneck, load imbalance, poor scalability
6.4.5 Method 2: nn.DistributedDataParallel (DDP)
Explains how DDP trains independent model replicas on separate GPUs and synchronizes their gradients without a central GPU bottleneck.
- Independent replicas and data shards: Describe how DDP gives each GPU its own process, model replica, optimizer, and unique slice of the batch, then has each process compute a local loss and gradients.
Why it matters: This explains how DDP spreads training work across GPUs without assigning one GPU the central workload. - Gradient averaging and synchronized updates: Trace how DDP averages locally computed gradients across peers, overlaps communication with backward computation, and lets every process apply the same update to keep model replicas synchronized.
Why it matters: This is why separate GPU processes can train one synchronized model without a boss GPU collecting gradients and updating weights.
Key terms: DistributedDataParallel, DDP, model replica, sharded, process, All-Reduce, Ring-AllReduce, averaged gradient, backward pass, optimizer step
6.4.6 Practical implementation of DDP
Explains the practical setup for running PyTorch DistributedDataParallel (DDP) training across GPUs.
- DDP process and device setup: Describe how torchrun launches one process per GPU, and how the training script initializes communication and assigns each process to its GPU.
Why it matters: These steps give each GPU its own process and let the processes coordinate training. - DDP model and data integration: Explain why the model is wrapped in DistributedDataParallel, why a DistributedSampler is used, and what changes in the training loop.
Why it matters: The wrapper and sampler handle the two practical differences that make the familiar training loop work across processes.
Key terms: torchrun, rank, process group, LOCAL_RANK, DistributedDataParallel, DDP wrapper, All-Reduce, gradient synchronization, DistributedSampler, ddp_model
6.4.7 DDP + gradient accumulation
This section combines DDP and gradient accumulation to overcome training-time and per-GPU memory limits, then works through the batch-size calculation.
- Complementary training bottlenecks: Explain how DDP and gradient accumulation address different limits in large-scale training: DDP spreads computation across GPUs, while accumulation achieves an effective batch size too large to fit on one device.
Why it matters: The two techniques solve different constraints, so using them together provides a benefit neither gives alone. - Effective batch calculation: Apply the section’s example to calculate the global batch and accumulation steps, and describe when the accumulated gradients lead to an update.
Why it matters: The calculation connects GPU count, per-GPU batch size, and accumulation steps to the target effective batch.
Key terms: Distributed Data Parallelism, Gradient Accumulation, time bottleneck, memory bottleneck, effective batch size, physical batch, global batch size, accumulation steps, synchronized update
6.5 Model parallelism
Explains why data parallelism cannot train a model that exceeds one GPU’s memory and introduces model parallelism as the alternative.
- Data parallelism memory limit: Explain why DDP and gradient accumulation do not solve the problem of a model that cannot fit on one GPU.
Why it matters: This limit explains why adding GPUs for data parallelism is not enough when the model itself is too large. - Model parallelism core idea: Describe model parallelism as splitting the model itself across multiple GPUs rather than splitting the data.
Why it matters: It provides the basic distinction needed for the model-parallel techniques that follow.
Key terms: Data Parallelism, DDP, VRAM, optimizer states, Model Parallelism
6.5.1 Key terminology for model parallelism
Defines the terms used to describe model-parallel pipelines and judge their efficiency.
- Inter-GPU communication overhead: Explain what communication means in parallel computing and why advanced parallelism techniques try to reduce or overlap it with computation.
Why it matters: Communication cost helps explain why two ways of splitting work across GPUs can have different performance. - Pipeline stages and activation flow: Describe what a stage is in model parallelism and how activations move between stages.
Why it matters: Stages name the parts of the pipeline whose work and waiting time later schedules coordinate. - Micro-batch size and purpose: Describe a micro-batch and state why the section introduces it for pipeline parallelism.
Why it matters: Micro-batches are the smaller pieces of data used when discussing efficient pipeline schedules. - Pipeline bubble idle time: Explain what the pipeline bubble is, when GPUs experience it, and why advanced pipeline techniques seek to minimize it.
Why it matters: The bubble is the section’s measure of how much expensive GPU time a pipeline wastes.
Key terms: Communication, All-Reduce, Stage, activations, Micro-batch, Pipeline Parallelism, Bubble, Pipeline Bubble, forward pass, backward pass
6.5.2 Pipeline parallelism: The assembly line approach
Explains pipeline parallelism as dividing a model between Transformer layers so data passes through stages on different GPUs.
- Pipeline stages and data flow: Explain how pipeline parallelism partitions consecutive Transformer blocks into stages on different GPUs and passes data from one stage to the next, rather than splitting computation within a layer.
Why it matters: This stage-by-stage layout is the basis for understanding pipeline schedules.
Key terms: Pipeline Parallelism, Tensor Parallelism, stages, Transformer blocks
6.5.3 The naive schedule (GPipe) and the pipeline bubble
Explains how GPipe fills and drains a micro-batch pipeline and why the resulting idle time is called a pipeline bubble.
- Forward pipeline fill: Trace how GPipe feeds micro-batches through successive GPU stages and explain why later stages initially sit idle.
Why it matters: The fill phase shows why adding pipeline stages also introduces startup idle time. - Backward pipeline drain: Explain why GPipe's backward pass starts at the last stage and leaves earlier-stage GPUs waiting.
Why it matters: The reverse dependency explains the second source of idle GPU time in the schedule. - Combined pipeline bubble: Explain what the pipeline bubble represents in GPipe and why it worsens as the number of stages grows.
Why it matters: The bubble captures the GPU utilization cost of this naive schedule, especially with many stages.
Key terms: micro-batches, fill phase, activations, backward pass, gradients, drain phase, GPipe, pipeline bubble, pipeline stages
6.5.4 The 1F1B schedule: Eliminating the bubble
This section explains how 1F1B interleaves pipeline work and why its lower activation-memory use improves practical efficiency even though its theoretical bubble overhead matches GPipe’s.
- 1F1B pipeline phases: Trace the warm-up, steady-state, and cool-down phases of 1F1B, explaining when a pipeline stage switches between forward and backward work.
Why it matters: The schedule makes use of backward work that is ready instead of leaving a pipeline stage idle. - Fixed-size pipeline efficiency: Explain why 1F1B and GPipe have the same theoretical maximum utilization for a fixed number of micro-batches and stages, despite their different schedules.
Why it matters: It prevents the denser-looking schedule from being mistaken for a reduction in fixed-size theoretical bubble overhead. - Activation memory advantage: Explain how 1F1B reduces peak activation memory relative to GPipe and why that can improve real-world utilization.
Why it matters: The memory advantage explains why 1F1B can outperform GPipe in practice without changing the fixed-size theoretical efficiency.
Key terms: 1F1B, warm-up, steady state, cool-down, micro-batch, pipeline efficiency, bubble steps, GPipe, intermediate activations, peak memory
6.5.5 The DeepSeek innovation: DualPipe Parallelism
Explains why MoE communication remains a bottleneck under 1F1B and introduces the folded stage assignment that enables DualPipe scheduling.
- MoE communication bottleneck: Explain why a balanced 1F1B pipeline can still leave GPUs idle when training a large MoE model.
Why it matters: It explains why fixing the ordinary pipeline bubble is not enough for large MoE models. - Folded pipeline stage assignment: Describe how pipeline folding assigns model stages to GPUs differently from a standard sequential pipeline.
Why it matters: The stage layout is the structural idea needed to follow the DualPipe schedules.
Key terms: 1F1B, Mixture-of-Experts (MoE), token shuffling, MoE communication, pipeline folding, stages, ranks, DualPipe
6.5.6 The DualPipe schedule: Hiding MoE communication
Explains how DualPipe overlaps computation and MoE communication across micro-batches to hide the cost of All-to-All communication.
- Fine-grained DualPipe overlap: Explain how DualPipe schedules Attention, Expert computation, and All-to-All communication across micro-batches, and why this helps large MoE training.
Why it matters: The schedule helps keep computation productive while MoE data moves between devices.
Key terms: DualPipe, fine-grained computation-communication overlap, micro-batch, All-to-All communication, Attention, MLP (Expert)
6.5.7 A more practical implementation: The DualPipeV schedule
This section explains how DualPipeV folds an N-stage pipeline into a V-shaped path on N/2 GPUs, reducing pipeline idle time while increasing per-GPU parameter memory.
- V-shaped micro-batch path: Trace a micro-batch through the folded pipeline and describe how its backward pass follows that path in reverse.
Why it matters: The route explains how the pipeline fits two stages on each device. - Single-stream bubble reduction: Explain how DualPipeV simplifies the full bidirectional schedule and why it still reduces the pipeline bubble.
Why it matters: This distinguishes DualPipeV’s simpler scheduling from the peak efficiency of full DualPipe. - GPU and memory trade-off: Describe the hardware benefit of the folded pipeline and the parameter-memory cost on each GPU.
Why it matters: The schedule’s hardware savings are only useful if each GPU can hold its larger share of the model.
Key terms: DualPipeV, folded pipeline, downstroke, turnaround, upstroke, single, continuous stream, pipeline bubble, interleaving, full bidirectional schedule, N/2 physical GPUs
6.5.8 The trade-offs
This section compares how training-parallelism techniques address different bottlenecks and what each costs.
- Memory and hardware trade-offs: Explain the bottleneck and trade-off for gradient accumulation, DDP, and GPipe.
Why it matters: Choosing a technique depends on whether batch memory, training time, or model size is the limiting factor. - Pipeline scheduling trade-offs: Trace why 1F1B improves on naive pipelining and why MoE training motivates DualPipe or DualPipeV, including their remaining cost.
Why it matters: A schedule that reduces idle time can expose communication as the next bottleneck, so its memory cost also matters.
Key terms: Gradient Accumulation, Data Parallelism (DDP), Pipeline Parallelism (GPipe), pipeline bubble, 1F1B Schedule, activation-memory cost, All-to-All communication, DualPipe, DualPipeV
6.6 Summary
Where this is used later
- (chapter intro) (chapter 7) · The training pipeline produces the base model that is subsequently post-trained.
Concepts explained in this chapter
- pipeline bubbleA pipeline bubble is an interval in pipeline-parallel execution when one or more accelerator stages are idle because work has not yet reached them or has already left them.
- gradient accumulationAn accumulation step is one smaller-batch computation whose gradients contribute to a later optimizer update.
- bidirectional DualPipe scheduleDualPipe is a training scheduling technique that overlaps forward and backward computations from different micro-batches to keep accelerators efficiently utilized.
- batchingA micro-batch is a smaller subdivision of a training batch that can be processed as an individual computation unit.
- data parallelismDistributed data parallelism is a training technique that replicates a model across devices and processes different data batches on each replica with synchronized parameter updates.
- backpropagationA backward pass is the computation that propagates gradients backward through a neural network to support parameter optimization.
- pipeline parallelismPipeline parallelism is a model-parallel training technique that divides computation into sequential stages and processes multiple micro-batches through them.
- 1F1B scheduleA 1F1B schedule is a pipeline-parallel execution schedule that interleaves one forward pass with one backward pass across successive micro-batches.
- MiniDeepSeekMiniDeepSeek is the book's compact DeepSeek-style language model that integrates the described architecture components into a single PyTorch module.
- key-value cacheA key-value cache stores previously computed attention keys and values, or their compressed representations, for reuse during autoregressive decoding.
- transformer blockA transformer block is a repeated neural-network unit combining attention, a feed-forward network, normalization, and residual connections.
- all-to-all communicationAll-to-all communication is a distributed communication operation in which every participating device exchanges data with every other participating device.
- evaluation datasetA validation set is a held-out subset of data used to measure how well a model generalizes during development.
- training loopA training loop is an iterative procedure that processes batches, computes predictions and loss, backpropagates gradients, updates parameters, and clears gradients.
- optimizationAn optimizer is an algorithm that updates model parameters using gradients to minimize a training objective.
- forward passA forward pass is the computation that propagates input data through a neural network to produce model outputs.
- mixture of expertsDeepSeek-MoE is a mixture-of-experts feed-forward architecture that routes inputs among specialized expert networks.
- pipeline efficiencyPipeline efficiency is the fraction of total pipeline execution time spent performing computation rather than waiting in bubbles.
- DistributedDataParallelnn.DistributedDataParallel is a PyTorch module for synchronously training replicated models across multiple processes or devices.
- tiktokentiktoken is a fast tokenization library that implements byte-pair encoding for language-model text.
- DualPipeVDualPipeV is a pipeline-parallel training schedule that folds an N-stage pipeline into a V-shaped execution path using a single continuous stream of micro-batches.
- GPipeGPipe is a pipeline-parallel training schedule that divides a batch into micro-batches and executes them through model stages with a fill-and-drain pattern.
- DeepSeekAttentionDeepSeekAttention is a PyTorch module implementing the decoupled attention architecture described in this section.
- MTPModuleMTPModule is a module that transforms a prior hidden state and the next-token embedding to produce a hidden state for an additional prediction step.
- RotaryPositionalEncodingRotaryPositionalEncoding is a module that generates and applies rotary positional encodings to query and key representations.
- mixed precisionMixed precision is a computation technique that uses multiple numerical precisions to reduce resource use while maintaining training effectiveness.
- end-of-sequence tokenAn end-of-text token is a special token marking the boundary between documents or text sequences.
- MTP moduleAn MTP module is a neural-network component that predicts an additional future token from a representation produced at the preceding prediction depth.
- pipelineA training pipeline is an organized sequence of data preparation, model computation, and optimization operations for training a machine-learning model.
- FP32Float32, also called FP32, is a 32-bit floating-point numerical format that uses four bytes per value.
- datasetA dataset is a collection of data examples organized for analysis or for training, validating, or evaluating machine-learning models.
- routingA router is a small learnable neural network that assigns scores to experts for each input token in a mixture-of-experts model.
- linear layerA linear layer is a neural-network layer that maps input representations to outputs through a learned affine transformation.
- load imbalanceLoad imbalance is an uneven distribution of tokens or computational work across expert modules.
- decoupled rotary position embeddingDecoupled RoPE is a rotary positional-encoding arrangement that separates positional information from content representations in attention.
- multi-head latent attentionMulti-head latent attention is an attention mechanism that represents key-value information in a compressed latent space to reduce memory use while retaining model quality.
- multi-token predictionMulti-token prediction is a language-model training or inference technique that predicts multiple future tokens rather than only the next token.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
