liveBook · Manning

DeepSeekAttention

As explained in Build a DeepSeek Model (From Scratch)

DeepSeekAttention is a PyTorch module implementing the decoupled attention architecture described in this section.

DeepSeekAttention is a PyTorch module implementing decoupled Multi-Head Latent Attention with separate content and position pathways. The content path compresses key-value information into c_kv for cache efficiency; the position path applies RoPE to specialized key and query projections. Their attention scores are added before applying the resulting weights to content values. section 3.16 section 6.2.1

Build the module around these implementation stages:

  • Precompute RoPE frequencies and rotate paired query-key dimensions, optionally starting from a position offset during cached inference. section 3.16 section 6.2.1
  • Project inputs through the content path to obtain q_c, compressed c_kv, and reconstructed k_c and v_c tensors. section 3.16 section 6.2.1
  • Project specialized positional queries and keys, reshape them per head, and apply RoPE to form the position path. section 3.16 section 6.2.1
  • During inference, append new c_kv and rotated positional keys to past_kv, then return the context output with the updated cache. section 6.2.1

Cached single-token inference

Pass a new token representation together with past_kv; derive the position offset from the cached length, rotate the new positional query and key at that offset, concatenate the new latent content with cached content, and return the updated cache. section 6.2.1

Figure 3.36 A detailed flowchart of the DeepSeekAttention module. The input tensor x is processed through two parallel streams: the blue Content Path, which uses pure MLA for cache efficiency, and the purple Position Path, which applies RoPE. The two cached tensors, c_kv and k_r, are highlighted with key icons. The outputs are combined to produce the final context vector.

Use the flowchart to trace the input through parallel content and position streams, identify the cached c_kv and k_r tensors, and follow their combination into the final context vector. section 3.16

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Descends from architectural pathways to PyTorch initialization, tensor reshaping, RoPE application, score combination, and cache updates.
Examples
Uses a forward-pass flowchart, RotaryPositionalEncoding helper, module initialization, and forward-method listings as implementation artifacts.
What it assumes
Assumes the MLA and RoPE foundations needed to translate their mathematical definitions into tensor projections and rotations.
What you can do after
Build it from scratch in PyTorch with decoupled content and position paths plus cached inference support.
How it connects
Connects Multi-Head Latent Attention, rotary positional encoding, attention masks, KV caching, and Transformer blocks.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.