liveBook · Manning

Chapter 1: Introduction to DeepSeek

What you will learn in chapter 1 of Build a DeepSeek Model (From Scratch)

The introduction frames the book as a hands-on, scaled-down reconstruction of DeepSeek rather than an attempt to reproduce its production-scale training run.

After this chapter you will be able to

Section by section

Introduction

The introduction frames the book as a hands-on, scaled-down reconstruction of DeepSeek rather than an attempt to reproduce its production-scale training run.

Key terms: educational, scaled-down DeepSeek clone, production-scale DeepSeek training run

1.1 Why DeepSeek? A turning point in open-source AI

This section explains why DeepSeek is a useful case study: it narrowed the gap with proprietary models while publishing enough detail for others to study and reproduce its methods.

Key terms: proprietary, open-source, LLaMA, DeepSeek-R1, DeepSeek-V3, reasoning benchmarks, open weights, technical reports, data and training pipelines

1.2 The key innovations we will build

Introduces a roadmap for studying DeepSeek innovations by examining the bottlenecks they address and then building replacements from scratch.

1.2.1 Architecture

This section contrasts a standard Transformer layer with DeepSeek’s architectural replacements and explains the problems and trade-offs associated with four DeepSeek innovations.

Key terms: multi-head self-attention, feed-forward neural network, Transformer, Multi-Head Latent Attention (MLA), DeepSeek-Mixture-of-Experts (MoE), compressed KV cache, routing, load-balancing, Multi-Token Prediction (MTP), FP8 quantization

1.2.2 Training

This section introduces DualPipe as a training schedule that keeps hardware busy by overlapping forward and backward computation.

Key terms: DualPipe, forward pass, backward pass, micro-batch, prefetching

1.2.3 Post-training

Introduces the post-training stages that turn DeepSeek-V3 into DeepSeek-R1 and the use of distillation to produce smaller models.

Key terms: DeepSeek-V3, DeepSeek-R1, cold-start, reinforcement learning, rejection sampling, synthetic training data, knowledge distillation, model compression, teacher, student

1.3 Book structure and scope

This section outlines the book’s four-stage roadmap, progressing from LLM inference foundations through DeepSeek architecture, training, and post-training.

1.4 What this book will teach you and what it won’t

This section defines the book’s hands-on, first-principles approach and distinguishes it from reproducing or deploying DeepSeek at full scale.

Key terms: proprietary training data, exact model weights, production deployment, clarity and comprehension, first principles, standard attention, Multi-Head Latent Attention (MLA)

1.5 What you will need to follow along

This section sets out the background and computing resources needed to follow the book and explains how its examples remain accessible.

Key terms: scaled-down versions, smaller datasets, consumer hardware, Python, backpropagation, PyTorch, transformer architecture, CPU, consumer GPU, VRAM

1.6 Summary

Where this is used later

Concepts explained in this chapter

Chapter 2: Solving the inference bottleneck with the key-value cache →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.