MTPModule extends causal prediction by transforming a previous hidden state together with the next-token embedding into a refined hidden state for one additional prediction step. section 6.2.1
Implement the step as two operations:
- Concatenate h_prev and next_token_embeds, then project the resulting 2d_model vector back to d_model. section 6.2.1
- Process the projected sequence with a TransformerBlock under an upper-triangular causal attention mask. section 6.2.1 section 6.2.1
- Return the block output as h_k; omit KV caching inside MTP during training. section 6.2.1
Forward path
For tensors shaped B × S × d_model, concatenate along the feature dimension, project to B × S × d_model, apply the causal mask, and return h_k. section 6.2.1
Listing 6.14 The Causal Multi-Token Prediction Module
class MTPModule(nn.Module):
def __init__(self, args: ModelArgs):
super().__init__()
self.projection = nn.Linear(args.d_model * 2, args.d_model, bias=False) #A
self.block = TransformerBlock(args) #B
def forward(self, h_prev, next_token_embeds):
x = torch.cat([h_prev, next_token_embeds], dim=-1)
x = self.projection(x)
B, S, D = x.shape
# Create a causal mask for this step
mask = torch.triu(torch.ones(S, S, device=x.device, dtype=torch.bool), diagonal=1)
attn_mask = torch.zeros(S, S, device=x.device).masked_fill(mask, float('-inf'))
# We don't use KV cache inside MTP during training
h_k, _ = self.block(x, attn_mask=attn_mask.unsqueeze(0).unsqueeze(0)) #C
return h_kInspect the complete PyTorch implementation of the causal multi-token prediction step. section 6.2.1
