liveBook · Manning

Causation chain

As explained in Build a DeepSeek Model (From Scratch)

A causal chain is an ordered sequence of prediction stages in which each stage conditions on the hidden state produced by the preceding stage.

A causation chain is a sequential multi-token prediction architecture: each prediction stage conditions on the refined hidden state produced by the preceding stage. For depth D=3, three chained MTP modules predict successive future tokens, unlike independent output heads that never exchange intermediate representations. section 5.3.2 section 5.3

Figure 5.8 The sequential architecture of the DeepSeek MTP modules. The hidden state from one module is passed as an input to the next, forming a causal chain.

Trace the hidden state from the main Transformer trunk through successive MTP modules; each module refines the representation before the next prediction. section 5.3.2

Each module combines two inputs: the previous hidden state hᵏ⁻¹ and the embedding of the future token at offset k. Separate RMSNorm layers normalize them before concatenation and projection. A Transformer block then produces the refined state, while a shared un-embedding matrix projects it into vocabulary logits. section 5.3.2

A functional implementation assembles the chain in stages:

  • Implement RMSNorm for numerical stability, then define a causal MTP module with two normalization layers, a 2d-to-d projection, and one Transformer block. section 5.4
  • Run the forward pass sequentially, slicing the usable sequence length at each depth and combining the main loss with losses from the MTP predictions. section 5.4

Three-step verification

With vocabulary size 1,000, model width 128, sequence length 20, batch size 2, and D=3, the implementation produces main logits of [2,20,1000], followed by [2,19,1000], [2,18,1000], and [2,17,1000] for the three chain depths.

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains inputs, normalization, projection, Transformer refinement, vocabulary projection, sequence slicing, and combined loss, but not large-scale training integration.
Examples
Demonstrates D=3 using PyTorch modules, random token sequences, depth-dependent logits, and a reported combined loss.
What it assumes
Assumes familiarity with Transformer trunks, hidden states, embeddings, vocabulary logits, and PyTorch module construction.
What you can do after
Build it from scratch in PyTorch with chained modules, shared parameters, sequential logits, and combined training losses.
How it connects
Connects multi-token prediction with Transformer blocks, hidden states, embeddings, RMSNorm, shared output layers, and teacher forcing.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.