liveBook · Manning

What you will learn in Build a DeepSeek Model (From Scratch)

by Raj Abhijit Dandekar, Rajat Dandekar, Sreedath Panat, Naman Dwivedi · Manning Early Access Program (MEAP)

Chapter by chapter

Chapter 1: Introduction to DeepSeek

The introduction frames the book as a hands-on, scaled-down reconstruction of DeepSeek rather than an attempt to reproduce its production-scale training run.

1.1 Why DeepSeek? A turning point in open-source AI · 1.2 The key innovations we will build · 1.2.1 Architecture · 1.2.2 Training · 1.2.3 Post-training · 1.3 Book structure and scope · 1.4 What this book will teach you and what it won’t · 1.5 What you will need to follow along · 1.6 Summary

What you learn in chapter 1 →

Chapter 2: Solving the inference bottleneck with the key-value cache

Introduces the chapter’s roadmap from inefficient autoregressive inference through KV caching and its memory cost to MQA and GQA as context for DeepSeek’s later architecture.

2.1 The LLM inference loop: Generating text one token at a time · 2.1.1 Distinguishing pretraining from inference · 2.1.2 Autoregressive process: Appending tokens to build context · 2.1.3 Visualizing autoregressive generation with GPT-2 · 2.2 The core task: Predicting the next token · 2.2.1 From input embeddings to context vectors: A mathematical walkthrough · 2.2.2 From context vectors to logits · 2.2.3 The key insight: Why only the last row matters · 2.3 The problem of redundant computations · 2.3.1 Intuition: Are we calculating the same thing over and over? · 2.3.2 A mathematical proof: Visualizing repeated calculations · 2.3.3 The performance effect: From quadratic to linear complexity · 2.4 The solution: Caching for efficiency · 2.4.1 What to cache? A step-by-step derivation · 2.4.2 The new inference loop with KV caching · 2.4.3 Demonstrating the speedup of KV caching · 2.5 The dark side of the KV cache: The memory cost · 2.5.1 The KV cache formula: Deconstructing the size · 2.5.2 The scaling problem in practice · 2.6 The memory-first approach: Multi-Query Attention (MQA) · 2.6.1 The core idea: Sharing a single key and value projection · 2.6.2 The impact on the KV cache formula · 2.6.3 The performance trade-off: Loss of expressivity · 2.6.4 Implementing an MQA layer from scratch · 2.7 The middle ground: Grouped-Query Attention (GQA) · 2.7.1 The core idea: Sharing keys and values within groups · 2.7.2 The tunable knob: Balancing memory and performance · 2.7.3 Implementing a GQA layer from scratch · 2.8 The performance vs. memory trade-off · 2.9 Summary

What you learn in chapter 2 →

Chapter 3: The DeepSeek breakthrough: Multi-Head Latent Attention (MLA)

The chapter introduction frames MLA as a response to the KV-cache memory trade-off and explains why positional awareness remains necessary.

3.1 MLA: The best of both worlds · 3.2 The MLA architecture: A visual walkthrough · 3.2.1 The query path (unchanged) · 3.2.2 The key/value path (the innovation) · 3.3 The mathematical magic: How the latent matrix helps · 3.3.1 A Step-by-Step Derivation of Q, K, and V in MLA · 3.3.2 The absorption trick: How attention scores are calculated · 3.3.3 The final step: Calculating the context vector · 3.4 The new inference loop with MLA · 3.4.1 What happens when a new token arrives? · 3.4.2 Caching the latent vector: The only thing we store · 3.4.3 Decompressing the cache and calculating attention · 3.5 Quantifying the gains · 3.5.1 The new KV cache formula: A 64x reduction · 3.5.2 Preserving performance: Why head diversity is maintained · 3.6 Building an MLA module from scratch · 3.7 The problem of order · 3.8 Attempt #1: The naive approach - integer positional encodings · 3.8.1 The simple idea: Using position numbers directly · 3.8.2 The major flaw: Polluting semantic embeddings with large magnitudes · 3.9 Attempt #2: A step forward - Binary positional encodings · 3.9.1 Solving the magnitude problem with binary representation · 3.9.2 Uncovering a deeper pattern: Oscillation frequencies · 3.9.3 The new problem: The issue with discontinuous jumps · 3.10 Attempt #3: The "Attention Is All You Need" breakthrough - sinusoidal positional encodings · 3.10.1 From discrete jumps to smooth waves: Introducing sine and cosine · 3.10.2 The power of rotation: Encoding relative positions · 3.10.3 The remaining flaw: Still polluting the token embeddings · 3.11 The state-of-the-art: Rotary Positional Encoding (RoPE) · 3.11.1 The core insights: Injecting position into attention and preserving magnitude · 3.11.2 The mechanism: Rotating query and key vectors · 3.12 The new challenge: Why standard RoPE and MLA don't mix · 3.13 The incompatibility problem: Why standard MLA and RoPE don’t work together · 3.14 The DeepSeek solution: Decoupled rotary position embedding · 3.14.1 The new architecture: A visual and mathematical deep dive · 3.14.2 Combining the paths to calculate final attention · 3.14.3 The cost of decoupling: A trade-off between memory and computation · 3.14.4 The new inference loop in action: What happens when a new token arrives · 3.15 Quantifying the gains: The final cache memory comparison · 3.16 Building MLA + decoupled RoPE from scratch · 3.17 The Payoff: An Empirical Head-to-Head Comparison · 3.18 Summary

What you learn in chapter 3 →

Chapter 4: Mixture-of-Experts (MoE) in DeepSeek: Scaling intelligence efficiently

Introduces MoE as an older idea whose use in large language models is more recent, and places DeepSeek’s approach in that history.

4.1 The intuition behind mixture of experts · 4.1.1 The problem with dense FFNs in transformer: High parameter count and computational cost · 4.1.2 The sparsity solution: Activating only a subset of experts per token · 4.1.3 Expert specialization: The "why" behind sparsity · 4.2 The mechanics of MoE: A hands-on mathematical walkthrough · 4.2.1 The goal: Combining multiple expert outputs into one · 4.2.2 Sparsity in action: Top-K selection for load balancing · 4.2.3 The routing mechanism: From input to expert scores · 4.2.4 From scores to weights: Top-K selection and softmax normalization · 4.2.5 The final output: Creating the weighted sum of expert outputs · 4.3 The challenge of balance: Ensuring all experts contribute · 4.3.1 Attempt #1: The auxiliary loss · 4.3.2 Attempt #2: The load balancing loss · 4.3.3 Attempt #3: A hard cap—the capacity factor · 4.4 The DeepSeek innovations: Towards ultimate expert specialization · 4.4.1 Core problems with traditional MoE · 4.4.2 Innovation #1: Fine-grained expert segmentation · 4.4.3 Innovation #2: Shared expert isolation · 4.4.4 Innovation #3: Auxiliary-loss-free load balancing · 4.5 Building a complete DeepSeek-MoE language model from scratch · 4.6 The payoff: An empirical head-to-head comparison · 4.7 Summary

What you learn in chapter 4 →

Chapter 5: Multi-token prediction and FP8 quantization

This chapter introduction shifts from DeepSeek’s architecture to its training techniques and distinguishes FP8 use in training from its earlier use in inference.

5.1 The core idea: From single-token to multi-token prediction · 5.2 The four key advantages of MTP · 5.2.1 Densification of training signals · 5.2.2 Improved data efficiency · 5.2.3 Better planning by prioritizing "choice points" · 5.2.4 Higher inference speed via speculative decoding · 5.3 The DeepSeek MTP architecture: A visual and mathematical walkthrough · 5.3.1 The starting point: The shared transformer trunk · 5.3.2 The MTP modules: A sequential chain of prediction · 5.3.3 The final loss calculation · 5.4 Implementing a causal multi-token prediction module from scratch · 5.5 Quantization: Trading precision for speed and memory · 5.5.1 What is quantization? · 5.5.2 Why quantize? The memory cost of high-precision parameters · 5.5.3 Understanding numerical formats: The building blocks of quantization · 5.5.4 The basic mechanism: Scaling · 5.6 The five pillars of DeepSeek's FP8 training · 5.6.1 Pillar 1: The mixed precision framework · 5.6.2 Pillar 2: Fine-grained quantization · 5.6.3 Pillar 3: Increasing accumulation precision · 5.6.4 Pillar 4: Mantissa over exponents · 5.6.5 Pillar 5: Online quantization · 5.7 Summary

What you learn in chapter 5 →

Chapter 6: The DeepSeek training pipeline: Building a foundation model

The chapter introduction previews assembling earlier DeepSeek components into a trainable mini-model and later examining DualPipe for large-scale training.

6.1 The data foundation: Preparing the TinyStories dataset · 6.1.1 Choosing the right tools: The TinyStories dataset and TikToken · 6.1.2 Setting up the environment · 6.1.3 The prepare.py script: A step-by-step walkthrough · 6.2 Assembling the Mini-DeepSeek model · 6.2.1 Building the components: RoPE, MLA, MoE, and MTP · 6.2.2 The final architecture: MiniDeepSeek · 6.3 The training pipeline: Bringing the model to life · 6.3.1 Configuration and system setup · 6.3.2 Loading data and scheduling the learning rate · 6.3.3 The main training script · 6.4 The engine of scale: Understanding DualPipe Parallelism · 6.4.1 The standard training loop and its memory limit · 6.4.2 Gradient accumulation in code · 6.4.3 Data parallelism: The "more GPUs, less time" approach · 6.4.4 Method 1: nn.DataParallel (DP) · 6.4.5 Method 2: nn.DistributedDataParallel (DDP) · 6.4.6 Practical implementation of DDP · 6.4.7 DDP + gradient accumulation · 6.5 Model parallelism · 6.5.1 Key terminology for model parallelism · 6.5.2 Pipeline parallelism: The assembly line approach · 6.5.3 The naive schedule (GPipe) and the pipeline bubble · 6.5.4 The 1F1B schedule: Eliminating the bubble · 6.5.5 The DeepSeek innovation: DualPipe Parallelism · 6.5.6 The DualPipe schedule: Hiding MoE communication · 6.5.7 A more practical implementation: The DualPipeV schedule · 6.5.8 The trade-offs · 6.6 Summary

What you learn in chapter 6 →

Chapter 7: Reinforcement learning: From policy gradients to GRPO

Introduces reinforcement-learning post-training as a way to turn a text-predicting base model toward rewarded reasoning behavior, using DeepSeek-R1 as the example.

7.1 The reinforcement learning framework · 7.1.1 The agent-environment interface · 7.1.2 Mapping reinforcement learning to language generation · 7.1.3 Sparse rewards and credit assignment · 7.2 Policy-gradient methods: Updating the LLM with rewards · 7.2.1 Sampling actions instead of taking argmax · 7.2.2 The basic policy-gradient update · 7.2.3 Why raw rewards are too noisy · 7.3 PPO: The standard practical baseline · 7.3.1 PPO components for LLM training · 7.3.2 Where the value function comes from · 7.3.3 Why the reference model matters · 7.4 GRPO: DeepSeek's value model simplification · 7.4.1 Group-relative advantages · 7.4.2 The full GRPO objective · 7.4.3 Why GRPO is cheaper than PPO · 7.5 Reinforcement learning with verifiable rewards · 7.5.1 Designing a verifier · 7.5.2 Avoiding overstatement · 7.6 How reasoning emerges in DeepSeek-R1-Zero · 7.7 DeepSeek-R1, R1-Zero, and Distill · 7.7.1 From R1-Zero to R1 · 7.7.2 Why cold-start data matters · 7.7.3 Rejection sampling as data construction · 7.7.4 Restoring general assistant behavior · 7.7.5 The final hybrid reward stage · 7.7.6 Distillation into smaller models · 7.8 Minimal chapter code · 7.8.1 A minimal verifier · 7.8.2 Computing group-relative advantages · 7.8.3 The clipped GRPO loss · 7.8.4 One training-step skeleton · 7.9 A worked GRPO example · 7.9.1 What happens to each token? · 7.9.2 Why multiple samples for the same prompt matter · 7.10 From toy code to a real LLM trainer · 7.10.1 Response format and verifier reliability · 7.10.2 trainiMonitoring behavior · 7.10.3 What changes from PPO to GRPO in code? · 7.11 What does this chapter add to the book? · 7.12 A practical implementation checklist · 7.13 Reading DeepSeek-R1 claims carefully · 7.14 Common implementation pitfalls · 7.15 Summary

What you learn in chapter 7 →

Chapter 8: Knowledge distillation: Making powerful models practical

The introduction motivates knowledge distillation as a way to make DeepSeek-R1’s reasoning ability practical to deploy in much smaller models.

8.1 Why 671 billion parameters won’t fit in your pocket · 8.1.1 The cost of intelligence · 8.1.2 The dream: Reasoning in your pocket · 8.2 The teacher-student paradigm · 8.2.1 Why soft labels carry more information than hard labels · 8.2.2 The master chef analogy · 8.3 Temperature and dark knowledge · 8.3.1 The temperature-scaled softmax · 8.3.2 A numerical walkthrough: seeing the dark knowledge emerge · 8.3.3 What the dark knowledge reveals · 8.3.4 Why T² scaling matters · 8.3.5 The temperature Goldilocks zone · 8.4 Building the distillation loss: From naive to powerful · 8.4.1 Attempt #1: Training from scratch with hard labels · 8.4.2 Attempt #2: Matching the teacher’s output at T=1 · 8.4.3 Temperature-scaled soft targets · 8.4.4 The complete distillation loss · 8.5 DeepSeek-R1’s distillation recipe · 8.5.1 From logits to language: a paradigm shift · 8.5.2 The 800K training dataset: Rejection sampling at scale · 8.5.3 Classical KD versus DeepSeek’s approach · 8.5.4 The six distilled models · 8.5.5 The economics of distillation · 8.6 Implementing knowledge distillation in PyTorch · 8.6.1 Defining teacher and student models · 8.6.2 The training loop · 8.6.3 Comparing the three approaches · 8.7 The payoff: An empirical head-to-head comparison · 8.7.1 The headline: 1.5B beats GPT-4o · 8.7.2 The critical comparison: Distillation versus direct RL · 8.7.3 Scaling behavior · 8.8 The limits of distillation · 8.8.1 The capacity gap · 8.8.2 The knowledge frontier · 8.8.3 The reproducibility gap · 8.8.4 Looking ahead · 8.9 Summary

What you learn in chapter 8 →

Concepts explained in this book

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.