An additional head extends next-token prediction by forecasting one further future token. In DeepSeek-style multi-token prediction, heads are chained rather than independent: each head predicts a token and produces a refined hidden state for the next head. section 5.3.2
A chained prediction head performs two coupled operations:
- Receives the previous head's hidden state and the embedding of its target future token. section 5.3.2
- Normalizes, merges, and projects those inputs before applying a Transformer block that refines the representation. section 5.3.2
- Projects the refined state through a shared un-embedding matrix to obtain vocabulary logits and select the predicted token. section 5.3.2
Prediction depth D=3
Set up three chained MTP heads. Head 1 predicts the next token, passes its refined state to Head 2, and Head 2 passes its state to Head 3. The number of heads therefore defines the prediction depth. Other implementations may reuse one lightweight head across depths, so the chain is an architectural choice rather than the only design. section 5.3.2
Figure 5.8 The sequential architecture of the DeepSeek MTP modules. The hidden state from one module is passed as an input to the next, forming a causal chain.

Use the sequential architecture diagram to trace how each head passes a refined hidden state into the next prediction step. section 5.3.2
Figure 5.12 The final prediction steps within an MTP module. The new hidden state generated by the MTP Transformer block is passed to a "Shared Un-Embedding Matrix." This projects the hidden state into the vocabulary space to produce a logits vector, from which the final token for that prediction step is selected.

Use the final-step diagram to inspect the projection from a refined hidden state into vocabulary logits. section 5.3.2
