liveBook · Manning

MTP module

As explained in Build a DeepSeek Model (From Scratch)

An MTP module is a neural-network component that predicts an additional future token from a representation produced at the preceding prediction depth.

An MTP module extends next-token prediction into a causal chain. Each module predicts one additional future token and produces a refined hidden state for the next prediction depth. A depth of D=3 therefore uses three chained modules rather than independent output heads. section 5.3 section 5.3.2

At depth k, the module receives the previous hidden state hᵏ⁻¹ and the embedding of the future token it must predict. It normalizes and concatenates these vectors, projects the result, runs a Transformer block, and maps the new hidden state into vocabulary logits through a shared un-embedding matrix. section 5.3.2

A practical training loop follows this sequence:

  • Run the main Transformer trunk and retain its normalized hidden states as h_main. section 6.2.2
  • Provide each module with ground-truth future-token embeddings through teacher forcing, not sampled predictions. section 5.3
  • Use the refined output as h_prev for the next module and combine the auxiliary losses with the main next-token loss. section 5.3.2

Depth-three chain

For D=3, module 1 predicts t+1 from h⁰ and Emb(t+1); module 2 predicts t+2 from the refined state and Emb(t+2); module 3 predicts t+3 from the next refined state. At sequence boundaries, the loop stops when insufficient future tokens remain. section 5.3.2

The chain can also act as a draft model for speculative decoding: MTP modules propose several tokens, while the main model verifies the chunk in one forward pass. Implementations differ; DeepSeek-V3 uses separate modules, whereas other designs reuse a lightweight head across depths. section 5.2.4 section 5.3.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains inputs, normalization, projection, Transformer refinement, logits, teacher forcing, loss flow, and sequence-boundary handling.
Examples
Illustrates a depth-three chain, tensor slicing near sequence boundaries, and a visual hidden-state flow across modules.
What it assumes
Assumes familiarity with hidden states, token embeddings, Transformer blocks, vocabulary logits, and cross-entropy training.
What you can do after
Build it from scratch as a chained PyTorch module integrated with a Transformer training path.
How it connects
Connects multi-token prediction with Transformer blocks, teacher forcing, shared embeddings, speculative decoding, and KV caching.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.