What you will learn in Build a DeepSeek Model (From Scratch)
Chapter by chapter
Chapter 1: Introduction to DeepSeek
The introduction frames the book as a hands-on, scaled-down reconstruction of DeepSeek rather than an attempt to reproduce its production-scale training run.
What you learn in chapter 1 →Chapter 2: Solving the inference bottleneck with the key-value cache
Introduces the chapter’s roadmap from inefficient autoregressive inference through KV caching and its memory cost to MQA and GQA as context for DeepSeek’s later architecture.
What you learn in chapter 2 →Chapter 3: The DeepSeek breakthrough: Multi-Head Latent Attention (MLA)
The chapter introduction frames MLA as a response to the KV-cache memory trade-off and explains why positional awareness remains necessary.
What you learn in chapter 3 →Chapter 4: Mixture-of-Experts (MoE) in DeepSeek: Scaling intelligence efficiently
Introduces MoE as an older idea whose use in large language models is more recent, and places DeepSeek’s approach in that history.
What you learn in chapter 4 →Chapter 5: Multi-token prediction and FP8 quantization
This chapter introduction shifts from DeepSeek’s architecture to its training techniques and distinguishes FP8 use in training from its earlier use in inference.
What you learn in chapter 5 →Chapter 6: The DeepSeek training pipeline: Building a foundation model
The chapter introduction previews assembling earlier DeepSeek components into a trainable mini-model and later examining DualPipe for large-scale training.
What you learn in chapter 6 →Chapter 7: Reinforcement learning: From policy gradients to GRPO
Introduces reinforcement-learning post-training as a way to turn a text-predicting base model toward rewarded reasoning behavior, using DeepSeek-R1 as the example.
What you learn in chapter 7 →Chapter 8: Knowledge distillation: Making powerful models practical
The introduction motivates knowledge distillation as a way to make DeepSeek-R1’s reasoning ability practical to deploy in much smaller models.
What you learn in chapter 8 →Concepts explained in this book
- distillationKnowledge distillation is a model-training technique in which a student model learns to reproduce information or behavior provided by a teacher model.
- mixture of expertsDeepSeek-MoE is a mixture-of-experts feed-forward architecture that routes inputs among specialized expert networks.
- key-value cacheA key-value cache stores previously computed attention keys and values, or their compressed representations, for reuse during autoregressive decoding.
- multi-head latent attentionMulti-head latent attention is an attention mechanism that represents key-value information in a compressed latent space to reduce memory use while retaining model quality.
- latent kv matrixA latent KV cache stores compressed key-value representations for previously processed tokens so attention can reuse historical context without storing full key and value states.
- expert networkAn expert is a specialized neural subnetwork within a mixture-of-experts model that processes tokens selected by a routing mechanism.
- student modelA distilled model is a student model trained to reproduce capabilities or behavior transferred from a larger teacher model.
- routingA router is a small learnable neural network that assigns scores to experts for each input token in a mixture-of-experts model.
- multi-head attentionMulti-head self-attention computes attention over an input sequence through multiple parallel attention heads, with each head having its own learned projections.
- dark knowledgeDark knowledge is the informative structure in a teacher model's output probabilities beyond the most likely prediction, including relationships among alternative outputs.
- FP8An 8-bit floating-point numerical format used to represent and compute neural-network values with reduced precision.
- multi-token predictionMulti-token prediction is a language-model training or inference technique that predicts multiple future tokens rather than only the next token.
- Group Relative Policy OptimizationGRPO, or Group Relative Policy Optimization, is a reinforcement-learning algorithm that estimates relative advantages among grouped sampled responses and optimizes a policy with a clipped objective and regularization.
- reinforcement learningReinforcement learning is a machine-learning paradigm in which an agent improves its behavior by optimizing rewards received from interactions or evaluations.
- Rotary Position EmbeddingRotary positional encoding is a positional-encoding technique that applies position-dependent rotations to query and key representations while preserving their magnitudes.
- context vectorA context vector is a numerical representation summarizing information relevant to a model's prediction at a particular sequence position.
- teacher modelA teacher model is a model whose outputs or learned behavior provide supervision for training another model.
- multi-query attentionMulti-query attention uses multiple query heads that share a smaller set of key and value projections to reduce attention-memory requirements.
- positional embeddingPositional encoding is a technique that represents a token's sequence position and incorporates that information into its semantic representation.
- FP32Float32, also called FP32, is a 32-bit floating-point numerical format that uses four bytes per value.
- transformer blockA transformer block is a repeated neural-network unit combining attention, a feed-forward network, normalization, and residual connections.
- pipeline bubbleA pipeline bubble is an interval in pipeline-parallel execution when one or more accelerator stages are idle because work has not yet reached them or has already left them.
- keyA key is an attention representation against which a query is compared to determine the relevance of a sequence position.
- reward signalA scalar reward is a single numerical value representing the quality or desirability of an agent's completed response or action.
- queryA query vector is an attention representation that specifies what information a token seeks from other tokens.
- attentionStandard attention is an attention mechanism that computes interactions between query representations and key-value representations to combine relevant information.
- expert loadExpert load is the number or proportion of input tokens routed to each expert during an MoE computation.
- temperatureTemperature is a scalar parameter that controls the sharpness of a probability distribution produced from model logits.
- shared expertA shared expert is an expert subnetwork in a mixture-of-experts architecture that is available across routed inputs rather than selected as a mutually exclusive expert.
- context matrixA context matrix contains attention outputs for all sequence positions, with each row formed by weighting value vectors.
- advantage functionAn advantage is a reinforcement-learning signal representing how much better or worse an action's outcome is than a reference baseline.
- bidirectional DualPipe scheduleDualPipe is a training scheduling technique that overlaps forward and backward computations from different micro-batches to keep accelerators efficiently utilized.
- valueA value is an attention representation containing the information contributed by a token when its key is selected.
- softmaxSoftmax is a function that converts a vector of scores into nonnegative values summing to one.
- mixed precisionMixed precision is a computation technique that uses multiple numerical precisions to reduce resource use while maintaining training effectiveness.
- soft targetA soft target is a target distribution that assigns graded values or probabilities to multiple possible classes rather than selecting only one class.
- fine-tuningFine-tuning is continued training of a pretrained model on a targeted dataset to adapt its behavior or capabilities.
- attention scoreAn attention score is a scalar measuring the relevance between a query and a key before normalization into attention weights.
- linear layerA linear layer is a neural-network layer that maps input representations to outputs through a learned affine transformation.
- gradient accumulationAn accumulation step is one smaller-batch computation whose gradients contribute to a later optimizer update.
- attention weightAn attention weight is a context-dependent coefficient that determines how strongly an input representation contributes to an attention output.
- policyA policy is a decision rule that assigns action probabilities or actions to situations encountered by a reinforcement-learning agent.
- Kullback–Leibler divergenceKL divergence is a measure of the discrepancy between two probability distributions that quantifies information lost when one distribution approximates another.
- backpropagationA backward pass is the computation that propagates gradients backward through a neural network to support parameter optimization.
- load violationLoad violation is a signed measure of an expert's deviation from the desired routing load, distinguishing under-utilization from over-utilization.
- grouped-query attentionGrouped-query attention partitions query heads into groups whose members share key and value projections while different groups retain distinct projections.
- forward passA forward pass is the computation that propagates input data through a neural network to produce model outputs.
- autoregressive modelAutoregressive generation is a text-generation process that predicts one token at a time and feeds each generated token back into the context for the next prediction.
- DeepSeekDeepSeek is a family of large language models developed with specialized attention, expert-routing, prediction, and post-training methods.
- absorption trickThe absorption trick is an algebraic attention optimization that pre-combines fixed query and key projection matrices so attention scores can be computed from a compressed latent representation.
- DeepSeek-R1DeepSeek-R1 is a reasoning-focused large language model designed to achieve strong performance on demanding reasoning tasks.
- stateA state is the information describing the current situation from which a reinforcement-learning policy selects an action.
- batchingA micro-batch is a smaller subdivision of a training batch that can be processed as an individual computation unit.
- dynamic bias adjustmentDynamic bias adjustment is a feedback-based technique that changes per-expert routing biases according to measured routing loads.
- routing matrixA routing matrix is a learned matrix that projects token representations into one routing score for each expert.
- probability distributionA probability distribution assigns nonnegative probabilities to possible outcomes, with the probabilities summing to one.
- generationGeneration is the process of producing a sequence of tokens from a language model, typically by sampling or selecting successive outputs.
- pipelineA training pipeline is an organized sequence of data preparation, model computation, and optimization operations for training a machine-learning model.
- routed expertA routed expert is an expert subnetwork that receives tokens selectively through the mixture-of-experts routing mechanism.
- MTP moduleAn MTP module is a neural-network component that predicts an additional future token from a representation produced at the preceding prediction depth.
- embeddingAn embedding is a learned vector representation of a discrete input such as a token, including its semantic information.
- bfloat16BF16 is a 16-bit floating-point numerical format with a wide exponent range commonly used for machine-learning computation.
- auxiliary-loss-free balancingAuxiliary-loss-free balancing is an expert-routing method that promotes balanced expert utilization without adding an auxiliary balancing loss.
- reference modelA reference policy is a fixed or separately maintained policy whose output probabilities provide a comparison distribution during policy optimization.
- fine-grained quantizationFine-grained quantization reduces numerical precision independently within small groups of activations or blocks of weights, assigning each region its own scaling factor.
- sparsitySparsity is a computational property in which only a subset of possible model connections, components, or parameters is active for a given computation.
- data parallelismDistributed data parallelism is a training technique that replicates a model across devices and processes different data batches on each replica with synchronized parameter updates.
- rejection samplingRejection sampling is a data curation procedure that generates multiple candidates and retains only those meeting specified correctness and quality criteria.
- value functionA value function estimates the expected return associated with a state, action, or policy in reinforcement learning.
- expressive powerRepresentational capacity is the ability of a model architecture to represent functions, patterns, or input-output relationships.
- content pathA content path is an attention-processing path that derives query, key, and value representations from semantic token content without positional encoding.
- position pathA position path is an attention-processing path that derives query and key representations carrying positional information separately from semantic content.
- decoupled rotary position embeddingDecoupled RoPE is a rotary positional-encoding arrangement that separates positional information from content representations in attention.
- pipeline parallelismPipeline parallelism is a model-parallel training technique that divides computation into sequential stages and processes multiple micro-batches through them.
- MiniDeepSeekMiniDeepSeek is the book's compact DeepSeek-style language model that integrates the described architecture components into a single PyTorch module.
- dequantizationDequantization is the transformation of quantized values back into a higher-precision numerical representation using quantization scales.
- dynamic rangeDynamic range is the span between the smallest and largest magnitudes that a numerical representation or computation can represent effectively.
- feedforward networkA feed-forward neural network transforms its inputs through layers whose connections do not form recurrent cycles.
- reward modelA reward model is a learned model that scores outputs according to preference-based or otherwise non-mechanical judgments of desired behavior.
- causation chainA causal chain is an ordered sequence of prediction stages in which each stage conditions on the hidden state produced by the preceding stage.
- matrix-matrix multiplicationMatrix multiplication is an algebraic operation that combines rows of one matrix with columns of another to produce a matrix.
- policy gradient methodA policy-gradient loss is an optimization objective that weights policy log probabilities by feedback-derived advantages to update a policy.
- verifiable rewardA verifiable reward is a training reward whose correctness can be determined by an executable checker, test, or other objective procedure.
- model compressionModel compression is the reduction of a machine-learning model's size or computational requirements while attempting to preserve its behavior.
- load-balancing lossA load-balancing loss is an auxiliary objective that encourages routing assignments to be distributed evenly across mixture-of-experts modules.
- DeepSeekAttentionDeepSeekAttention is a PyTorch module implementing the decoupled attention architecture described in this section.
- decoupled attention architectureA decoupled attention architecture separates semantic-content processing from relative-position processing and fuses their attention scores.
- 1F1B scheduleA 1F1B schedule is a pipeline-parallel execution schedule that interleaves one forward pass with one backward pass across successive micro-batches.
- relative positionA relative positional relationship describes the positional difference between tokens rather than representing each absolute position independently.
- training loopA training loop is an iterative procedure that processes batches, computes predictions and loss, backpropagates gradients, updates parameters, and clears gradients.
- chain-of-thought promptingChain-of-thought is a sequence of intermediate natural-language reasoning steps produced between a problem statement and its final answer.
- post-trainingPost-training is the stage of adapting a pretrained model through additional optimization, data generation, or alignment procedures.
- DeepSeek-R1-Distill-Qwen-1.5BDeepSeek-R1-Distill-Qwen-1.5B is a 1.5-billion-parameter language model fine-tuned on reasoning data generated from DeepSeek-R1.
- expert specializationExpert specialization is the division of a model's processing capacity among expert modules that learn different input patterns or functions.
- masked attentionA causal mask is an attention constraint that prevents each token from accessing later tokens in the sequence.
- logitA logit is an unnormalized real-valued score assigned to a possible output token before conversion to probabilities.
- log probabilityA log probability is the logarithm of the probability assigned to an event, such as a generated token under a language-model policy.
- baselineA baseline model is a reference model trained under a simpler or comparison-only procedure without teacher guidance.
- datasetA dataset is a collection of data examples organized for analysis or for training, validating, or evaluating machine-learning models.
- scaling lawA scaling law is an empirical relationship describing how a model's performance or other properties change as resources such as model size increase.
- all-to-all communicationAll-to-all communication is a distributed communication operation in which every participating device exchanges data with every other participating device.
- anomalyAn outlier is a numerical activation, weight, or gradient whose magnitude is unusually large relative to other values in its group.
- gating weightA gating weight is a normalized routing coefficient that determines an input token's contribution from a selected expert.
- routing biasA routing bias is a persistent per-expert score adjustment that influences expert selection without being learned through backpropagation.
- knowledge hybridityKnowledge hybridity is the condition in which an MoE expert must represent diverse unrelated knowledge and skills because the model has too few experts for its training data.
- precisionPrecision is the fineness with which a numerical format represents significant digits and distinguishes nearby values.
- knowledge redundancyKnowledge redundancy is the duplication of common information across multiple model components that could otherwise specialize in different functions.
- auxiliary-loss-free bias updateAn auxiliary-loss-free bias update adjusts expert routing biases from observed load deviations without adding an auxiliary balancing loss.
- dynamic routing biasA dynamic routing bias is a non-trainable per-expert adjustment to router scores that is updated to improve load balance.
- routed pathA routed path is the sparse MoE computation in which each token is processed by a selected subset of experts.
- shared pathA shared path is the dense MoE computation in which every token is processed by a common set of experts.
- master weightA master weight is a high-precision persistent copy of a neural-network parameter used as the source for optimization updates.
- sparse dispatchSparse dispatch batches tokens by assigned expert, computes expert outputs, and combines the outputs at their original token positions.
- top-k gatingTop-k gating selects the highest-scoring experts for each token and assigns normalized routing weights to them.
- weight gradientA weight gradient is the derivative of a training loss with respect to a neural-network weight parameter.
- language modelA neural language model that learns statistical patterns from large text corpora to generate and understand human language.
- feature importance scoreExpert importance is the aggregate selection weight assigned to an expert across the tokens in a batch.
- verifierA deterministic checker is a procedure that evaluates an output against fixed criteria and produces the same result for the same input.
- accumulatorAn accumulator is a hardware register that stores a running sum of products during a computation.
- gpu coreA CUDA core is a general-purpose arithmetic processing unit in an NVIDIA GPU that supports a broad range of computations.
- promotion to CUDA CoresPromotion to CUDA Cores is a mixed-precision accumulation technique that periodically transfers partial sums from low-precision Tensor Core computation to high-precision CUDA Core computation.
- FP32 registerAn FP32 register is a hardware register capable of storing a 32-bit floating-point value.
- Tensor CoreA Tensor Core is a GPU computational unit specialized for high-throughput matrix multiplication and accumulation, typically at reduced numerical precision.
- WGMMAWGMMA is an NVIDIA GPU instruction that performs warp-group-level matrix multiply-accumulate operations.
- KL regularizationKL control is the regulation of policy updates by penalizing divergence between the learned policy and a reference policy.
- underflowUnderflow is the condition in which a numerical value is too small in magnitude for a format to represent its detail accurately.
- quantizationQuantization is the process of representing higher-precision numerical values with lower-precision values, typically by scaling and rounding them.
- chain-of-thoughtReasoning behavior is a model's capability to carry out or express multi-step inference and problem-solving strategies.
- parameter countParameter count is the number of learned numerical parameters in a machine-learning model.
- scalar quantizationStandard quantization maps an entire tensor to a lower-precision numerical range using one shared scaling factor.
- pipeline efficiencyPipeline efficiency is the fraction of total pipeline execution time spent performing computation rather than waiting in bubbles.
- evaluation datasetA validation set is a held-out subset of data used to measure how well a model generalizes during development.
- additional headAn output head is the final neural-network component that transforms hidden representations into task-specific output scores or predictions.
- total lossTotal loss is an aggregate training error formed by summing the individual losses associated with multiple prediction targets.
- binary positional encodingBinary positional encoding represents sequence positions using binary-valued positional codes rather than unbounded integer values.
- reinforcement learning with verifiable rewardsReinforcement learning with verifiable rewards is reinforcement learning in which a deterministic procedure evaluates outputs and supplies the reward.
- attention headAn attention head is an independently parameterized attention submodule that computes relationship-specific attention within a multi-head attention mechanism.
- load imbalanceLoad imbalance is an uneven distribution of tokens or computational work across expert modules.
- expert routingRouting is the mechanism that selects which expert subnetworks process a given input in a mixture-of-experts architecture.
- auxiliary lossAn auxiliary loss is an additional penalty term optimized alongside a model's main training loss to encourage a desired property such as balanced expert selection.
- 2d rotationA 2D rotation is a geometric transformation that rotates a two-dimensional vector by an angle while preserving its length.
- exponentAn exponent is the component of a floating-point representation that determines the scale and dynamic range of represented values.
- mantissaA mantissa, or significand, is the component of a floating-point representation that determines the significant digits and precision of a value.
- top-kTop-k selection retains the k highest-scoring experts for each token and discards the others.
- hard-label lossHard-label loss is the cross-entropy error between a student's predicted token or class distribution and the ground-truth label.
- soft-target lossSoft-target loss is a divergence-based error between the probability distributions produced by a teacher and a student after temperature adjustment.
- DistributedDataParallelnn.DistributedDataParallel is a PyTorch module for synchronously training replicated models across multiple processes or devices.
- actionAn action is a choice made by an agent at a decision point that can change the state of an environment.
- latent spaceA latent space is a lower-level representational space in which compressed or abstract features encode information from the original representations.
- DeepSeek-R1-ZeroDeepSeek-R1-Zero is a reasoning-focused language model trained to develop reasoning behavior through reinforcement learning without conventional supervised fine-tuning as its initial reasoning-training stage.
- E4M3E4M3 is an FP8 encoding with four exponent bits and three explicitly stored mantissa bits.
- tiktokentiktoken is a fast tokenization library that implements byte-pair encoding for language-model text.
- merged embeddingA merged embedding is a vector formed by concatenating normalized hidden-state and token-embedding representations.
- un-embedding matrixAn un-embedding matrix is a learned output projection that maps hidden representations to scores over a model's vocabulary.
- distillation lossA distillation loss is a training objective that measures or combines discrepancies between a student's predictions and information supplied by a teacher model.
- DeepSeek distillation datasetThe DeepSeek distillation dataset is a book-specific collection of approximately 800,000 curated reasoning and non-reasoning training samples.
- Proximal Policy OptimizationProximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that constrains policy updates using a clipped or otherwise limited probability ratio.
- reasoning patternA reasoning pattern is a recurring strategy for carrying out multi-step inference or problem solving.
- end-of-sequence tokenAn end-of-text token is a special token marking the boundary between documents or text sequences.
- foundation modelA base model is a pretrained model whose existing parameters and capabilities provide the foundation for subsequent adaptation or distillation.
- logits matrixA logits matrix is a matrix of routing scores assigning each token a score for each available expert.
- optimizationAn optimizer is an algorithm that updates model parameters using gradients to minimize a training objective.
- standard moeStandard MoE is the book's baseline mixture-of-experts model using conventional auxiliary-loss-based load balancing.
- data explorationExploration is the process of trying varied actions or strategies to discover which ones produce desirable outcomes during learning.
- fine-grained expert segmentationFine-grained expert segmentation is a mixture-of-experts design that replaces a small pool of large experts with a larger pool of smaller, more specialized experts while preserving aggregate capacity.
- DualPipeVDualPipeV is a pipeline-parallel training schedule that folds an N-stage pipeline into a V-shaped execution path using a single continuous stream of micro-batches.
- GPipeGPipe is a pipeline-parallel training schedule that divides a batch into micro-batches and executes them through model stages with a fill-and-drain pattern.
- cold-start datasetA cold-start dataset is a relatively small initial dataset of curated examples used to establish a model's behavior before later optimization.
- shared expert isolationShared expert isolation is a mixture-of-experts design that separates dense shared experts for common knowledge from sparsely routed experts for specialized knowledge.
- MTPModuleMTPModule is a module that transforms a prior hidden state and the next-token embedding to produce a hidden state for an additional prediction step.
- RotaryPositionalEncodingRotaryPositionalEncoding is a module that generates and applies rotary positional encodings to query and key representations.
- rotationA rotation is a geometric transformation that changes the orientation of a vector by a specified angle while preserving its length.
- key up-projectorA key up-projector is a learned linear transformation that reconstructs attention key vectors from a latent key-value representation.
- value up-projectorA value up-projector is a learned linear transformation that reconstructs attention value vectors from a latent key-value representation.
- online quantizationOnline quantization calculates quantization scaling factors from the current tensor or batch during processing rather than from historical statistics.
- parameterA model parameter is a learned numerical value that determines part of a machine-learning model's computation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
