liveBook · Manning

Decoupled rotary position embedding

As explained in Build a DeepSeek Model (From Scratch)

Decoupled RoPE is a rotary positional-encoding arrangement that separates positional information from content representations in attention.

Decoupled rotary position embedding separates attention into parallel content and position paths. The content path handles semantic relevance through pure Multi-Head Latent Attention (MLA), while the position path applies RoPE to specialized queries and keys. Adding both score streams preserves positional reasoning without disrupting MLA's absorption-based cache optimization. section 3.14 section 3.16

The arrangement proceeds through four implementation stages:

  • Project each input into a compressed content latent, then derive content queries, keys, and values from that representation. section 3.16
  • Project specialized positional queries and keys into a separate RoPE dimension, keeping positional rotation outside the content path. section 3.16
  • Rotate the positional vectors, add content-based and position-based attention scores, and apply the resulting weights to content-only values. section 3.14 section 3.16
  • Cache the compressed content latent and rotated positional key rather than separate full key and value matrices. section 3.15

Cache comparison at DeepSeek-V2 scale

With 128 heads of size 128, standard MHA has a cache dimension of 32,768. Using d_latent = 512 and d_rope = 64 gives MLA with decoupled RoPE a dimension of 576, approximately 57 times smaller. The reported comparison also gives MLA the lowest validation loss, 2.3089, and perplexity, 10.06, while using 12 MB of KV-cache memory. section 3.15 section 3.17

Size_MLA+SizeRoPE = l * b * (d_latent + d_rope) * s * 2

Use the cache-size equation to track how the latent content dimension and decoupled positional dimension replace the larger MHA key-value storage term. section 3.15

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains the two-path calculation, cache dimensions, and PyTorch module structure, but does not detail the complete forward listing.
Examples
Uses a cache-dimension calculation, a DeepSeek-V2-scale comparison, PyTorch modules, and matched MHA, MQA, GQA, and MLA experiments.
What it assumes
Assumes familiarity with attention, RoPE, and MLA while grounding their interaction through projections, caches, and score combination.
What you can do after
Build it from scratch as a PyTorch attention module with separate content and RoPE position paths and combined attention scores.
How it connects
Connects RoPE with Multi-Head Latent Attention, attention scores, compressed latent caches, and KV-cache optimization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.