liveBook · Manning

MTPModule

As explained in Build a DeepSeek Model (From Scratch)

MTPModule is a module that transforms a prior hidden state and the next-token embedding to produce a hidden state for an additional prediction step.

MTPModule extends causal prediction by transforming a previous hidden state together with the next-token embedding into a refined hidden state for one additional prediction step. section 6.2.1

Implement the step as two operations:

  • Concatenate h_prev and next_token_embeds, then project the resulting 2d_model vector back to d_model. section 6.2.1
  • Process the projected sequence with a TransformerBlock under an upper-triangular causal attention mask. section 6.2.1 section 6.2.1
  • Return the block output as h_k; omit KV caching inside MTP during training. section 6.2.1

Forward path

For tensors shaped B × S × d_model, concatenate along the feature dimension, project to B × S × d_model, apply the causal mask, and return h_k. section 6.2.1

Listing 6.14 The Causal Multi-Token Prediction Module
class MTPModule(nn.Module):
    def __init__(self, args: ModelArgs):
        super().__init__()
        self.projection = nn.Linear(args.d_model * 2, args.d_model, bias=False) #A
        self.block = TransformerBlock(args) #B
    def forward(self, h_prev, next_token_embeds):
        x = torch.cat([h_prev, next_token_embeds], dim=-1)
        x = self.projection(x)
        B, S, D = x.shape
        # Create a causal mask for this step
        mask = torch.triu(torch.ones(S, S, device=x.device, dtype=torch.bool), diagonal=1)
        attn_mask = torch.zeros(S, S, device=x.device).masked_fill(mask, float('-inf'))        
        # We don't use KV cache inside MTP during training
        h_k, _ = self.block(x, attn_mask=attn_mask.unsqueeze(0).unsqueeze(0)) #C
        return h_k

Inspect the complete PyTorch implementation of the causal multi-token prediction step. section 6.2.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Defines projection, causal masking, TransformerBlock execution, and training-time KV-cache omission, while simplifying the full design.
Examples
Uses a PyTorch forward method that concatenates h_prev with next_token_embeds and applies an upper-triangular attention mask.
What it assumes
Assumes hidden states, token embeddings, TransformerBlock interfaces, and causal attention-mask construction.
What you can do after
Build it from scratch as a PyTorch module for one causal multi-token prediction step.
How it connects
Connects hidden states, token embeddings, causal masking, TransformerBlock, and KV caching.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.