A causation chain is a sequential multi-token prediction architecture: each prediction stage conditions on the refined hidden state produced by the preceding stage. For depth D=3, three chained MTP modules predict successive future tokens, unlike independent output heads that never exchange intermediate representations. section 5.3.2 section 5.3
Figure 5.8 The sequential architecture of the DeepSeek MTP modules. The hidden state from one module is passed as an input to the next, forming a causal chain.

Trace the hidden state from the main Transformer trunk through successive MTP modules; each module refines the representation before the next prediction. section 5.3.2
Each module combines two inputs: the previous hidden state hᵏ⁻¹ and the embedding of the future token at offset k. Separate RMSNorm layers normalize them before concatenation and projection. A Transformer block then produces the refined state, while a shared un-embedding matrix projects it into vocabulary logits. section 5.3.2
A functional implementation assembles the chain in stages:
- Implement RMSNorm for numerical stability, then define a causal MTP module with two normalization layers, a 2d-to-d projection, and one Transformer block. section 5.4
- Run the forward pass sequentially, slicing the usable sequence length at each depth and combining the main loss with losses from the MTP predictions. section 5.4
Three-step verification
With vocabulary size 1,000, model width 128, sequence length 20, batch size 2, and D=3, the implementation produces main logits of [2,20,1000], followed by [2,19,1000], [2,18,1000], and [2,17,1000] for the three chain depths.
