An MTP module extends next-token prediction into a causal chain. Each module predicts one additional future token and produces a refined hidden state for the next prediction depth. A depth of D=3 therefore uses three chained modules rather than independent output heads. section 5.3 section 5.3.2
At depth k, the module receives the previous hidden state hᵏ⁻¹ and the embedding of the future token it must predict. It normalizes and concatenates these vectors, projects the result, runs a Transformer block, and maps the new hidden state into vocabulary logits through a shared un-embedding matrix. section 5.3.2
A practical training loop follows this sequence:
- Run the main Transformer trunk and retain its normalized hidden states as h_main. section 6.2.2
- Provide each module with ground-truth future-token embeddings through teacher forcing, not sampled predictions. section 5.3
- Use the refined output as h_prev for the next module and combine the auxiliary losses with the main next-token loss. section 5.3.2
Depth-three chain
For D=3, module 1 predicts t+1 from h⁰ and Emb(t+1); module 2 predicts t+2 from the refined state and Emb(t+2); module 3 predicts t+3 from the next refined state. At sequence boundaries, the loop stops when insufficient future tokens remain. section 5.3.2
The chain can also act as a draft model for speculative decoding: MTP modules propose several tokens, while the main model verifies the chunk in one forward pass. Implementations differ; DeepSeek-V3 uses separate modules, whereas other designs reuse a lightweight head across depths. section 5.2.4 section 5.3.2
