liveBook · Manning

Multi-token prediction

As explained in Build a DeepSeek Model (From Scratch)

Multi-token prediction is a language-model training or inference technique that predicts multiple future tokens rather than only the next token.

Multi-token prediction (MTP) trains a language model to predict several future tokens from each position, rather than only the immediate next token. Each position receives feedback over a longer horizon, giving the model a denser training signal about sequence structure and coherence. section 5.1 section 5.2.1

A longer prediction horizon

Given “Artificial Intelligence is,” ordinary next-token prediction targets “transforming” at the position of “is.” MTP extends the targets to “transforming,” “the,” and “world,” with loss reflecting predictions across that future span. section 5.1

More feedback per example can improve data efficiency, but the benefit is not uniform. On MBPP and HumanEval, the reported advantage over single-token prediction became more consistent as models grew from 0.3B to 13B parameters; some very small models performed worse. Increasing the prediction horizon generally helped up to a point. section 5.2.2

Figure 5.4 Performance improvement of MTP over single-token prediction on coding benchmarks. Positive bars indicate MTP is better. (Source: Gloeckle et al., 2024)

Compare the reported MTP advantage over single-token prediction across model sizes. section 5.2.2

An initial MTP design used independent output heads: a head predicting the second future token had no information from the first prediction. DeepSeek instead chains MTP modules, each refining the preceding hidden state. During training, the chain uses ground-truth tokens through teacher forcing, not sampled predictions. section 5.3

A self-contained causal implementation can be built in stages:

  • Build an MTP module that combines the previous hidden state with an embedding of the next ground-truth token, then refines the result with a Transformer block. section 5.4
  • Wrap the main Transformer trunk and a sequence of MTP modules, sharing embedding and output layers across prediction depths. section 5.4
  • Run the chain at progressively shorter sequence lengths and combine its losses with the main prediction loss. section 5.4
Listing 5.4 The MTP Forward Pass and Combined Loss Calculation
# ... (inside the DeepSeekV3WithMTP class) ...
 
    def forward(self, input_ids: torch.Tensor, targets: Optional[torch.Tensor] = None):
        B, S = input_ids.shape
        # ... (other setup) ...
 
        # --- Main model forward pass ---
        x = self.get_embedding(input_ids)
        # ... (pass through self.blocks) ...
        h_main = self.norm_f(x)
        logits_main = self.get_output_logits(h_main)
 
        all_logits = [logits_main]
        h_prev = h_main #A
 
        # --- MTP chain: Sequential prediction ---
        for depth_k in range(1, self.num_mtp_heads + 1):
            L = S - depth_k #B
            if L <= 0: break
 
            h_prev_sliced = h_prev[:, :L, :] #C
 
            future_token_ids = input_ids[:, depth_k:depth_k + L]
            future_token_embeds = self.get_embedding(future_token_ids) #D
 
            h_curr = self.mtp_modules[depth_k - 1](h_prev_sliced, future_token_embeds) #E
            
            logits_k = self.get_output_logits(h_curr)
            all_logits.append(logits_k)
 
            h_prev = h_curr #F
 
        # --- Loss computation ---
        loss = None
        if targets is not None:
            # ... (loss calculation logic) ...
            total_loss = # ... Main model loss ...
 
            for k, logits_k in enumerate(all_logits[1:], start=1):
                # ... (calculate loss for MTP depth k) ...
                mtp_loss_sum += loss_mtp_k
            
            # Final loss: L = L_main + (λ/D) * Σ(L_MTP^k)
            if self.num_mtp_heads > 0 and mtp_loss_sum > 0:
                mtp_loss_weighted = (self.mtp_loss_weight / self.num_mtp_heads) * mtp_loss_sum #G
                total_loss += mtp_loss_weighted
            
            loss = total_loss
 
        return {"logits_all": all_logits, "loss": loss}

Inspect the sequential forward pass, including depth-dependent slicing and future-token embeddings. section 5.4

Checking output shapes

With batch size 2, sequence length 20, vocabulary size 1,000, and three MTP modules, the main logits have shape [2, 20, 1000]. The MTP logits shorten to [2, 19, 1000], [2, 18, 1000], and [2, 17, 1000] as prediction depth increases. section 5.4

MTP can also support speculative decoding: prediction heads draft candidate tokens, and the main model verifies the chunk in one forward pass. Speedups depend on the implementation. DeepSeek V3’s public release used standard single-token inference and discarded its MTP modules, although those modules could be repurposed for drafting. section 5.2.4 section 1.2.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains independent and causal designs, then implements a self-contained module, forward pass, and combined loss rather than a large-scale training integration.
Examples
Uses a sentence-completion example, coding benchmarks, and a three-module output-shape check.
What it assumes
Introduces next-token prediction before extending the objective to a multi-token horizon.
What you can do after
Build it from scratch in PyTorch as a self-contained causal MTP system.
How it connects
Connects MTP to next-token prediction, teacher forcing, Transformer blocks, and speculative decoding.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.