liveBook · Manning

Transformer block

As explained in Build a DeepSeek Model (From Scratch)

A transformer block is a repeated neural-network unit combining attention, a feed-forward network, normalization, and residual connections.

A transformer block turns token representations into context-aware representations through attention and a feed-forward network (FFN). Blocks are stacked: each block’s output becomes the next block’s input, letting the model refine representations before predicting the next token. section 2.2 section 2.2.2

Figure 2.3 High-level overview of the Transformer block architecture. The diagram shows the data flow from initial input tokens ("The next day is bright") through embeddings, multi-head attention, and feed-forward layers, producing the logits used for next-token prediction.

Follow tokens through embeddings, attention, and feed-forward layers toward next-token logits. section 2.2

A block combines four operations:

  • Multi-head attention relates tokens to one another, producing a context vector for each token. section 2.2 section 2.2.2
  • The FFN processes each context vector independently. It typically uses two linear layers with a nonlinear activation between them and preserves the input shape. section 2.2.2
  • Normalization and two residual connections complete the block. One residual connection adds the attention input to its output; the other does the same for the FFN. section 2.2.2

From context vectors to logits

For “The next day is,” attention produces a 4 × 4 context matrix in the small example. The FFN preserves that shape, and successive blocks refine the matrix. After the final block, a final normalization and output projection produce vocabulary logits for next-token prediction. section 2.2.2

One implementation normalizes before each sublayer: compute h, new_cache = attention(norm1(x), mask, past_kv, position_offset), then x = x + h, followed by x = x + feed_forward(norm2(x)). It returns both the updated representation and the attention cache. This version uses LayerNorm for simplicity, a DeepSeek-style attention module, and a mixture-of-experts FFN. section 6.2.1

Listing 6.15 The TransformerBlock Module
class TransformerBlock(nn.Module):
    def __init__(self, args: ModelArgs):
        super().__init__()
        # Use LayerNorm for simplicity, though DeepSeek uses RMSNorm
        self.norm1 = nn.LayerNorm(args.d_model) #A
        self.attention = DeepSeekAttention(args)
        self.norm2 = nn.LayerNorm(args.d_model) #B
        self.feed_forward = DeepSeekMoE(args)
    def forward(self, x: torch.Tensor, attn_mask: torch.Tensor, 
                past_kv: Optional[Tuple[torch.Tensor, torch.Tensor]] = None, 
                position_offset: int = 0):
        # Attention block with residual connection
        h, new_cache = self.attention(self.norm1(x), attn_mask, past_kv, position_offset) #C
        x = x + h
        # Feed-forward block with residual connection
        x = x + self.feed_forward(self.norm2(x)) #D
        return x, new_cache

Inspect the two normalization, sublayer, and residual steps in the TransformerBlock forward method. section 6.2.1

During cached inference, each block receives its own past cache and a position offset; the updated caches are returned for subsequent generation steps. A separate cache per layer means cache memory grows with the number of blocks. During training, the example model processes the full input sequence without using the KV cache. section 6.2.2 section 2.5.1

A block can also refine a hidden state inside a multi-token prediction module. In the DeepSeek V3 design, the previous hidden state and a future-token embedding are separately normalized, concatenated, and projected back to the model dimension before entering a dedicated block. Its output supplies a refined hidden state for the next prediction step. section 5.3.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains block data flow and residual structure, then implements a normalized attention and mixture-of-experts block with cache handling.
Examples
Traces a four-token context matrix to logits and implements a DeepSeek-style block in MiniDeepSeek.
What you can do after
Builds a transformer block from component modules and integrates it into a model stack with training and cached inference paths.
How it connects
Connects multi-head attention, feed-forward networks, residual connections, normalization, KV caching, and multi-token prediction.

Also explained in

Compare how 5 books explain transformer block →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.