Decoupled rotary position embedding separates attention into parallel content and position paths. The content path handles semantic relevance through pure Multi-Head Latent Attention (MLA), while the position path applies RoPE to specialized queries and keys. Adding both score streams preserves positional reasoning without disrupting MLA's absorption-based cache optimization. section 3.14 section 3.16
The arrangement proceeds through four implementation stages:
- Project each input into a compressed content latent, then derive content queries, keys, and values from that representation. section 3.16
- Project specialized positional queries and keys into a separate RoPE dimension, keeping positional rotation outside the content path. section 3.16
- Rotate the positional vectors, add content-based and position-based attention scores, and apply the resulting weights to content-only values. section 3.14 section 3.16
- Cache the compressed content latent and rotated positional key rather than separate full key and value matrices. section 3.15
Cache comparison at DeepSeek-V2 scale
With 128 heads of size 128, standard MHA has a cache dimension of 32,768. Using d_latent = 512 and d_rope = 64 gives MLA with decoupled RoPE a dimension of 576, approximately 57 times smaller. The reported comparison also gives MLA the lowest validation loss, 2.3089, and perplexity, 10.06, while using 12 MB of KV-cache memory. section 3.15 section 3.17
Use the cache-size equation to track how the latent content dimension and decoupled positional dimension replace the larger MHA key-value storage term. section 3.15
