Multi-token prediction (MTP) trains a language model to predict several future tokens from each position, rather than only the immediate next token. Each position receives feedback over a longer horizon, giving the model a denser training signal about sequence structure and coherence. section 5.1 section 5.2.1
A longer prediction horizon
Given “Artificial Intelligence is,” ordinary next-token prediction targets “transforming” at the position of “is.” MTP extends the targets to “transforming,” “the,” and “world,” with loss reflecting predictions across that future span. section 5.1
More feedback per example can improve data efficiency, but the benefit is not uniform. On MBPP and HumanEval, the reported advantage over single-token prediction became more consistent as models grew from 0.3B to 13B parameters; some very small models performed worse. Increasing the prediction horizon generally helped up to a point. section 5.2.2
Figure 5.4 Performance improvement of MTP over single-token prediction on coding benchmarks. Positive bars indicate MTP is better. (Source: Gloeckle et al., 2024)

Compare the reported MTP advantage over single-token prediction across model sizes. section 5.2.2
An initial MTP design used independent output heads: a head predicting the second future token had no information from the first prediction. DeepSeek instead chains MTP modules, each refining the preceding hidden state. During training, the chain uses ground-truth tokens through teacher forcing, not sampled predictions. section 5.3
A self-contained causal implementation can be built in stages:
- Build an MTP module that combines the previous hidden state with an embedding of the next ground-truth token, then refines the result with a Transformer block. section 5.4
- Wrap the main Transformer trunk and a sequence of MTP modules, sharing embedding and output layers across prediction depths. section 5.4
- Run the chain at progressively shorter sequence lengths and combine its losses with the main prediction loss. section 5.4
Listing 5.4 The MTP Forward Pass and Combined Loss Calculation
# ... (inside the DeepSeekV3WithMTP class) ...
def forward(self, input_ids: torch.Tensor, targets: Optional[torch.Tensor] = None):
B, S = input_ids.shape
# ... (other setup) ...
# --- Main model forward pass ---
x = self.get_embedding(input_ids)
# ... (pass through self.blocks) ...
h_main = self.norm_f(x)
logits_main = self.get_output_logits(h_main)
all_logits = [logits_main]
h_prev = h_main #A
# --- MTP chain: Sequential prediction ---
for depth_k in range(1, self.num_mtp_heads + 1):
L = S - depth_k #B
if L <= 0: break
h_prev_sliced = h_prev[:, :L, :] #C
future_token_ids = input_ids[:, depth_k:depth_k + L]
future_token_embeds = self.get_embedding(future_token_ids) #D
h_curr = self.mtp_modules[depth_k - 1](h_prev_sliced, future_token_embeds) #E
logits_k = self.get_output_logits(h_curr)
all_logits.append(logits_k)
h_prev = h_curr #F
# --- Loss computation ---
loss = None
if targets is not None:
# ... (loss calculation logic) ...
total_loss = # ... Main model loss ...
for k, logits_k in enumerate(all_logits[1:], start=1):
# ... (calculate loss for MTP depth k) ...
mtp_loss_sum += loss_mtp_k
# Final loss: L = L_main + (λ/D) * Σ(L_MTP^k)
if self.num_mtp_heads > 0 and mtp_loss_sum > 0:
mtp_loss_weighted = (self.mtp_loss_weight / self.num_mtp_heads) * mtp_loss_sum #G
total_loss += mtp_loss_weighted
loss = total_loss
return {"logits_all": all_logits, "loss": loss}Inspect the sequential forward pass, including depth-dependent slicing and future-token embeddings. section 5.4
Checking output shapes
With batch size 2, sequence length 20, vocabulary size 1,000, and three MTP modules, the main logits have shape [2, 20, 1000]. The MTP logits shorten to [2, 19, 1000], [2, 18, 1000], and [2, 17, 1000] as prediction depth increases. section 5.4
MTP can also support speculative decoding: prediction heads draft candidate tokens, and the main model verifies the chunk in one forward pass. Speedups depend on the implementation. DeepSeek V3’s public release used standard single-token inference and discarded its MTP modules, although those modules could be repurposed for drafting. section 5.2.4 section 1.2.1
