DeepSeekAttention is a PyTorch module implementing decoupled Multi-Head Latent Attention with separate content and position pathways. The content path compresses key-value information into c_kv for cache efficiency; the position path applies RoPE to specialized key and query projections. Their attention scores are added before applying the resulting weights to content values. section 3.16 section 6.2.1
Build the module around these implementation stages:
- Precompute RoPE frequencies and rotate paired query-key dimensions, optionally starting from a position offset during cached inference. section 3.16 section 6.2.1
- Project inputs through the content path to obtain q_c, compressed c_kv, and reconstructed k_c and v_c tensors. section 3.16 section 6.2.1
- Project specialized positional queries and keys, reshape them per head, and apply RoPE to form the position path. section 3.16 section 6.2.1
- During inference, append new c_kv and rotated positional keys to past_kv, then return the context output with the updated cache. section 6.2.1
Cached single-token inference
Pass a new token representation together with past_kv; derive the position offset from the cached length, rotate the new positional query and key at that offset, concatenate the new latent content with cached content, and return the updated cache. section 6.2.1
Figure 3.36 A detailed flowchart of the DeepSeekAttention module. The input tensor x is processed through two parallel streams: the blue Content Path, which uses pure MLA for cache efficiency, and the purple Position Path, which applies RoPE. The two cached tensors, c_kv and k_r, are highlighted with key icons. The outputs are combined to produce the final context vector.

Use the flowchart to trace the input through parallel content and position streams, identify the cached c_kv and k_r tensors, and follow their combination into the final context vector. section 3.16
