5 Multi-token prediction and FP8 quantization
This chapter covers
- Multi-Token Prediction for stronger training signals
- Implementing a causal MTP architecture
- Utilizing FP8 quantization to optimize training efficiency
We have now established the core architectural pillars of the DeepSeek model: Multi-Head Latent Attention and Mixture-of-Experts. These innovations define what the model computes. Now, we turn our attention to an equally important topic that defines how these computations are performed with incredible efficiency. This involves two key techniques that are central to DeepSeek's training methodology: Multi-Token Prediction (MTP) and FP8 Quantization. While FP8 quantization was already being adopted in the industry to accelerate inference, DeepSeek's key innovation was demonstrating its successful and stable application to the much more demanding task of large-scale training.
This chapter has two parts. First, we take a deep dive into MTP: its motivation, its advantages, and how DeepSeek built their advanced, causal version of it. You will learn the theory and also build a functional MTP module, seeing first-hand how predicting a horizon of tokens strengthens the model's planning ability. Second, we turn to the FP8 quantization framework that lets these massive models train with remarkable speed and memory efficiency.
Now let’s open the black box of these mechanisms. As illustrated in figure 5.1, our roadmap highlights the components we will build in this chapter.