Chapter 1: Introduction to DeepSeek
The introduction frames the book as a hands-on, scaled-down reconstruction of DeepSeek rather than an attempt to reproduce its production-scale training run.
After this chapter you will be able to
- Educational DeepSeek Clone Scope. Distinguish the book’s plan to build an educational, scaled-down DeepSeek clone from reproducing the full production-scale model and training run.
- DeepSeek-R1’s competitive performance. Explain why DeepSeek-R1 was presented as a turning point, including its relationship to DeepSeek-V3 and its reported performance relative to leading proprietary models.
- Open reporting and reproducibility. Distinguish DeepSeek’s openness from merely releasing weights, and explain why its published technical detail makes it valuable to learners and builders.
- DeepSeek Architectural Replacements. Explain which standard Transformer components DeepSeek replaces with MLA and DeepSeek-MoE, and identify the problem and trade-off associated with each.
- DualPipe computation overlap. Explain how DualPipe overlaps the forward pass of one micro-batch with the backward pass of another, and distinguish that overlap from ordinary prefetching.
- DeepSeek-R1 post-training sequence. Describe how DeepSeek-V3 becomes DeepSeek-R1 through cold-start fine-tuning, reinforcement learning, self-generated training data, and a final reinforcement learning phase.
- Teacher-student model distillation. Explain the purpose of distilling the large DeepSeek-R1 teacher into smaller student models.
- Practical reconstruction scope. Distinguish the book’s focus on understanding and implementing DeepSeek techniques from reproducing DeepSeek’s exact models or operating them at production scale.
- Accessible scaled-down experiments. Explain why the book’s implementations can be run without the resources used to train a full-scale language model.
Section by section
Introduction
The introduction frames the book as a hands-on, scaled-down reconstruction of DeepSeek rather than an attempt to reproduce its production-scale training run.
- Educational DeepSeek Clone Scope: Distinguish the book’s plan to build an educational, scaled-down DeepSeek clone from reproducing the full production-scale model and training run.
Why it matters: This sets realistic expectations for what building DeepSeek from scratch means in this book.
Key terms: educational, scaled-down DeepSeek clone, production-scale DeepSeek training run
1.1 Why DeepSeek? A turning point in open-source AI
This section explains why DeepSeek is a useful case study: it narrowed the gap with proprietary models while publishing enough detail for others to study and reproduce its methods.
- The earlier open-model gap: Describe the capability gap between open and proprietary language models before DeepSeek, and the shift toward releasing model weights that preceded it.
Why it matters: The earlier gap explains why DeepSeek’s results drew attention. - DeepSeek-R1’s competitive performance: Explain why DeepSeek-R1 was presented as a turning point, including its relationship to DeepSeek-V3 and its reported performance relative to leading proprietary models.
Why it matters: R1’s results made the narrowing gap between open and proprietary AI concrete. - Open reporting and reproducibility: Distinguish DeepSeek’s openness from merely releasing weights, and explain why its published technical detail makes it valuable to learners and builders.
Why it matters: The published detail is what makes building parts of DeepSeek a practical learning exercise rather than guesswork.
Key terms: proprietary, open-source, LLaMA, DeepSeek-R1, DeepSeek-V3, reasoning benchmarks, open weights, technical reports, data and training pipelines
1.2 The key innovations we will build
Introduces a roadmap for studying DeepSeek innovations by examining the bottlenecks they address and then building replacements from scratch.
1.2.1 Architecture
This section contrasts a standard Transformer layer with DeepSeek’s architectural replacements and explains the problems and trade-offs associated with four DeepSeek innovations.
- Standard Transformer Layer Components: Describe the two main components of a standard Transformer layer and what each does.
Why it matters: This baseline makes DeepSeek’s architectural changes clear. - DeepSeek Architectural Replacements: Explain which standard Transformer components DeepSeek replaces with MLA and DeepSeek-MoE, and identify the problem and trade-off associated with each.
Why it matters: These substitutions connect DeepSeek’s architecture to the scaling problems each component targets. - Additional Efficiency Techniques: Distinguish MTP from FP8 quantization by describing each technique’s stated purpose and trade-off.
Why it matters: MTP and FP8 address different efficiency problems from the two architectural replacements.
Key terms: multi-head self-attention, feed-forward neural network, Transformer, Multi-Head Latent Attention (MLA), DeepSeek-Mixture-of-Experts (MoE), compressed KV cache, routing, load-balancing, Multi-Token Prediction (MTP), FP8 quantization
1.2.2 Training
This section introduces DualPipe as a training schedule that keeps hardware busy by overlapping forward and backward computation.
- DualPipe computation overlap: Explain how DualPipe overlaps the forward pass of one micro-batch with the backward pass of another, and distinguish that overlap from ordinary prefetching.
Why it matters: The distinction explains why DualPipe can keep compute units busy beyond simply preparing data ahead of time.
Key terms: DualPipe, forward pass, backward pass, micro-batch, prefetching
1.2.3 Post-training
Introduces the post-training stages that turn DeepSeek-V3 into DeepSeek-R1 and the use of distillation to produce smaller models.
- DeepSeek-R1 post-training sequence: Describe how DeepSeek-V3 becomes DeepSeek-R1 through cold-start fine-tuning, reinforcement learning, self-generated training data, and a final reinforcement learning phase.
Why it matters: The sequence explains how post-training develops reasoning and broadens the base model’s usefulness. - Teacher-student model distillation: Explain the purpose of distilling the large DeepSeek-R1 teacher into smaller student models.
Why it matters: Distillation makes capabilities from a very large model available in smaller, more practical models.
Key terms: DeepSeek-V3, DeepSeek-R1, cold-start, reinforcement learning, rejection sampling, synthetic training data, knowledge distillation, model compression, teacher, student
1.3 Book structure and scope
This section outlines the book’s four-stage roadmap, progressing from LLM inference foundations through DeepSeek architecture, training, and post-training.
1.4 What this book will teach you and what it won’t
This section defines the book’s hands-on, first-principles approach and distinguishes it from reproducing or deploying DeepSeek at full scale.
- Practical reconstruction scope: Distinguish the book’s focus on understanding and implementing DeepSeek techniques from reproducing DeepSeek’s exact models or operating them at production scale.
Why it matters: This distinction sets realistic expectations for what building along with the book will accomplish. - First-principles building approach: Describe how the book proposes to develop understanding by building techniques step by step from simpler foundations.
Why it matters: The step-by-step approach is intended to help you modify techniques rather than merely copy an implementation.
Key terms: proprietary training data, exact model weights, production deployment, clarity and comprehension, first principles, standard attention, Multi-Head Latent Attention (MLA)
1.5 What you will need to follow along
This section sets out the background and computing resources needed to follow the book and explains how its examples remain accessible.
- Accessible scaled-down experiments: Explain why the book’s implementations can be run without the resources used to train a full-scale language model.
Why it matters: You can explore the ideas without reproducing the cost of training DeepSeek itself. - Required starting background: Describe the knowledge and programming experience expected before starting the book.
Why it matters: These prerequisites help you judge whether you are ready to work through the implementations. - Hardware and access options: Distinguish the hardware that can run most examples from hardware that makes experimentation faster or enables more ambitious work, and describe the offered setup support.
Why it matters: You can choose a practical way to follow along rather than assuming a high-end local GPU is mandatory.
Key terms: scaled-down versions, smaller datasets, consumer hardware, Python, backpropagation, PyTorch, transformer architecture, CPU, consumer GPU, VRAM
1.6 Summary
Where this is used later
- Assembling the Mini-DeepSeek model (chapter 6) · The book assembles a Mini-DeepSeek model from components studied in earlier chapters.
- MLA: The best of both worlds (chapter 3) · Provides the standard attention component against which MLA is compared.
- The intuition behind mixture of experts (chapter 4) · Provides the standard feed-forward component against which MoE is compared.
- The core idea: From single-token to multi-token prediction (chapter 5) · Develops the idea of predicting multiple tokens.
- Quantization: Trading precision for speed and memory (chapter 5) · Develops the precision and resource trade-off behind quantization.
- The DualPipe schedule: Hiding MoE communication (chapter 6) · The detailed DualPipe schedule builds on this introductory account of overlapping forward and backward work.
- From R1-Zero to R1 (chapter 7) · Revisits the progression from the base model to DeepSeek-R1.
- Rejection sampling as data construction (chapter 7) · Expands on rejection sampling as a way to construct training data.
Concepts explained in this chapter
- DeepSeekDeepSeek is a family of large language models developed with specialized attention, expert-routing, prediction, and post-training methods.
- mixture of expertsDeepSeek-MoE is a mixture-of-experts feed-forward architecture that routes inputs among specialized expert networks.
- multi-head latent attentionMulti-head latent attention is an attention mechanism that represents key-value information in a compressed latent space to reduce memory use while retaining model quality.
- FP8An 8-bit floating-point numerical format used to represent and compute neural-network values with reduced precision.
- multi-token predictionMulti-token prediction is a language-model training or inference technique that predicts multiple future tokens rather than only the next token.
- distillationKnowledge distillation is a model-training technique in which a student model learns to reproduce information or behavior provided by a teacher model.
- post-trainingPost-training is the stage of adapting a pretrained model through additional optimization, data generation, or alignment procedures.
- reinforcement learningReinforcement learning is a machine-learning paradigm in which an agent improves its behavior by optimizing rewards received from interactions or evaluations.
- DeepSeek-R1DeepSeek-R1 is a reasoning-focused large language model designed to achieve strong performance on demanding reasoning tasks.
- model compressionModel compression is the reduction of a machine-learning model's size or computational requirements while attempting to preserve its behavior.
- pipelineA training pipeline is an organized sequence of data preparation, model computation, and optimization operations for training a machine-learning model.
- language modelA neural language model that learns statistical patterns from large text corpora to generate and understand human language.
- key-value cacheA key-value cache stores previously computed attention keys and values, or their compressed representations, for reuse during autoregressive decoding.
- feedforward networkA feed-forward neural network transforms its inputs through layers whose connections do not form recurrent cycles.
- multi-head attentionMulti-head self-attention computes attention over an input sequence through multiple parallel attention heads, with each head having its own learned projections.
- rejection samplingRejection sampling is a data curation procedure that generates multiple candidates and retains only those meeting specified correctness and quality criteria.
- student modelA distilled model is a student model trained to reproduce capabilities or behavior transferred from a larger teacher model.
- teacher modelA teacher model is a model whose outputs or learned behavior provide supervision for training another model.
- generationGeneration is the process of producing a sequence of tokens from a language model, typically by sampling or selecting successive outputs.
- bidirectional DualPipe scheduleDualPipe is a training scheduling technique that overlaps forward and backward computations from different micro-batches to keep accelerators efficiently utilized.
- MiniDeepSeekMiniDeepSeek is the book's compact DeepSeek-style language model that integrates the described architecture components into a single PyTorch module.
- attentionStandard attention is an attention mechanism that computes interactions between query representations and key-value representations to combine relevant information.
- fine-tuningFine-tuning is continued training of a pretrained model on a targeted dataset to adapt its behavior or capabilities.
- cold-start datasetA cold-start dataset is a relatively small initial dataset of curated examples used to establish a model's behavior before later optimization.
- reasoning patternA reasoning pattern is a recurring strategy for carrying out multi-step inference or problem solving.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
