liveBook · Manning

Chapter 5: Multi-token prediction and FP8 quantization

What you will learn in chapter 5 of Build a DeepSeek Model (From Scratch)

This chapter introduction shifts from DeepSeek’s architecture to its training techniques and distinguishes FP8 use in training from its earlier use in inference.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This chapter introduction shifts from DeepSeek’s architecture to its training techniques and distinguishes FP8 use in training from its earlier use in inference.

Key terms: Multi-Head Latent Attention, Mixture-of-Experts, Multi-Token Prediction, FP8 Quantization, inference, large-scale training

5.1 The core idea: From single-token to multi-token prediction

This section contrasts standard next-token training with multi-token prediction and briefly explains how extra predictions can also speed up inference.

Key terms: Next-Token Prediction, Multi-Token Prediction, loss, speculative decoding, extra prediction heads, verification step

5.2 The four key advantages of MTP

Introduces the claim that predicting multiple tokens during training changes what and how efficiently the model learns, ahead of a discussion of four benefits.

5.2.1 Densification of training signals

Explains why predicting several future tokens from each position gives MTP denser training feedback than single-token prediction.

Key terms: MTP, single-token prediction, training signals, gradient signal

5.2.2 Improved data efficiency

This section explains how multi-token prediction (MTP) improves data efficiency and summarizes benchmark evidence, including limits to its gains.

Key terms: MTP, data efficiency, densification of training signals, single-token baseline, predicted future tokens, n, MBPP, HumanEval

5.2.3 Better planning by prioritizing "choice points"

Explains how multi-token prediction can improve planning by making consequential pattern-shifting tokens recur in the training loss.

Key terms: choice point, MTP loss, multi-token prediction, planning

5.2.4 Higher inference speed via speculative decoding

Explains how MTP heads could accelerate inference through speculative decoding, while distinguishing that possibility from how DeepSeek V3 was publicly released.

Key terms: speculative decoding, draft model, MTP heads, verification, forward pass, pre-training, standard single-token prediction, MTP modules

5.3 The DeepSeek MTP architecture: A visual and mathematical walkthrough

This section contrasts independent multi-token output heads with DeepSeek’s sequential MTP modules and clarifies how the prediction chain is formed during training.

Key terms: MTP modules, independent output heads, hidden state, causal chain, teacher forcing, ground-truth tokens

5.3.1 The starting point: The shared transformer trunk

This section identifies the shared Transformer trunk’s output, h⁰, as the starting point for multi-token prediction.

Key terms: Shared Transformer Trunk, hidden state, context vector, Hidden state 0, h⁰

5.3.2 The MTP modules: A sequential chain of prediction

This section traces how DeepSeek-V3 chains MTP modules, each combining a previous hidden state with a future-token embedding to produce a prediction and a refined state for the next module.

Key terms: MTP Modules, prediction depth, causal chain, refined hidden state, hᵏ⁻¹, ground-truth embedding, RMS Norm, Merged Embedding, Linear Projection Layer, Mₖ

5.3.3 The final loss calculation

This subsection explains how the losses for multiple future-token predictions are combined during training.

Key terms: D prediction depths, ground-truth tokens, cross-entropy losses, total loss

5.4 Implementing a causal multi-token prediction module from scratch

This section describes a self-contained PyTorch implementation of causal multi-token prediction, from its modules and shared layers to its forward pass, loss, and sequence-length check.

Key terms: DeepSeekMTPModule, main Transformer trunk, mtp_modules, shared embedding and output layers, h_main, h_prev, h_curr, prediction depth, MTP losses, causal chain

5.5 Quantization: Trading precision for speed and memory

Introduces FP8 quantization as a low-precision trade-off that helps make DeepSeek’s large-scale computation affordable.

Key terms: FP8 quantization, low precision, speed, memory efficiency

5.5.1 What is quantization?

This section defines quantization as representing model parameters with fewer bits and explains the resulting trade-off between precision, memory use, and performance.

Key terms: quantization, FP32, FP16, bit-width, precision

5.5.2 Why quantize? The memory cost of high-precision parameters

This section explains why quantization is useful: lower-precision parameters reduce model memory needs and can speed computation, at the cost of some precision.

Key terms: quantization, precision, GPU's main memory

5.5.3 Understanding numerical formats: The building blocks of quantization

This section explains how numerical formats allocate bits to range and precision, and why those choices matter for quantization.

Key terms: sign, exponent, mantissa, significand, dynamic range, precision, FP32, FP16, BF16, overflow

5.5.4 The basic mechanism: Scaling

This section explains how scaling maps FP32 values into a symmetric INT8 range and approximately recovers them afterward.

Key terms: scaling, Maximum Absolute Value (α), scaling factor, quantize, de-quantize, FP32

5.6 The five pillars of DeepSeek's FP8 training

This section introduces DeepSeek’s FP8 training as a five-pillar framework and previews the explanation of each pillar.

5.6.1 Pillar 1: The mixed precision framework

This section explains how DeepSeek uses FP8 for fast linear-layer matrix multiplications while retaining higher precision where training is sensitive to numerical error.

Key terms: mixed precision, FP8, FP32, BF16, GEMM, Fprop, Dgrad, activations, dL/dx, FP32 accumulator

5.6.2 Pillar 2: Fine-grained quantization

This section explains how DeepSeek uses separate scales for small groups of activations and blocks of weights to protect precision during low-precision training.

Key terms: tensor-wise scaling, outliers, scaling factor, quantize, Fine-Grained Quantization, activations, weights, groups, tiles, FP8

5.6.3 Pillar 3: Increasing accumulation precision

This section explains why long FP8 matrix multiplications can lose accuracy during accumulation and how DeepSeek combines Tensor Cores with FP32 accumulation on CUDA Cores to reduce that error.

Key terms: FP8, dot product, Tensor Cores, accumulator, Low Prec Acc, promotion to CUDA Cores, WGMMA, Nc interval, FP32 Register, Scaling Factors

5.6.4 Pillar 4: Mantissa over exponents

This section explains why DeepSeek uses the more precise E4M3 format throughout FP8 training instead of switching formats for backward passes.

Key terms: E5M2, E4M3, exponent, mantissa, dynamic range, precision, fine-grained quantization, outliers, scaling factors, forward pass

5.6.5 Pillar 5: Online quantization

This section explains why DeepSeek calculates FP8 scaling factors from the current tensor rather than relying on historical values.

Key terms: Delayed Quantization, scaling factor, historical maximum, overflow, underflow, Online Quantization, maximum absolute value, current tensor

5.7 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 4: Mixture-of-Experts (MoE) in DeepSeek: Scaling intelligence efficiently Chapter 6: The DeepSeek training pipeline: Building a foundation model →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.