liveBook · Manning

Position path

As explained in Build a DeepSeek Model (From Scratch)

A position path is an attention-processing path that derives query and key representations carrying positional information separately from semantic content.

A position path separates positional relevance from semantic relevance when standard Rotary Positional Encoding (RoPE) conflicts with Multi-Head Latent Attention (MLA). The content path keeps MLA position-agnostic so its absorption trick remains valid; the position path derives RoPE-encoded queries and keys. Their attention scores are added before applying the weights to content-only values. section 3.14 section 3.18

Implement the position path as a parallel stream:

  • Project the input X through a dedicated key matrix W_k_pos, apply RoPE, and share the resulting positional key projection across heads when possible. section 3.14.1
  • Reuse the latent query cQ from the content path, project it through W_q_pos, reshape it into heads, and apply RoPE to obtain positional queries. section 3.14.1
  • Add position-based scores to content-based scores, then use the combined attention weights with the content-only value matrix. section 3.16
Figure 3.29 The Position Path of the Decoupled Architecture. This new path creates specialized Query (Qr) and Key (Kr) matrices that are infused with Rotary Positional Encodings.

Use the architecture figure to track how RoPE-encoded Qr and Kr form a positional stream alongside content-based attention. section 3.14.1

A from-scratch PyTorch implementation groups separate linear layers under Content Path and Position Path headings. The forward pass reshapes each projection into attention heads, rotates only positional tensors, and retains the latent content tensor c_kv plus rotated key k_r for caching. section 3.16

Autoregressive update

When the next token is "bright," append its down-projected latent vector to the latent KV cache while computing its positional query and key through the RoPE path. Combine the new content and position scores to predict the following token. section 3.14.4

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Works through projections, reshaping, RoPE application, score addition, and caching, stopping short of complete source code.
Examples
Uses PyTorch attention listings, architecture flowcharts, and an autoregressive update for the token "bright."
What it assumes
Assumes familiarity with attention and MLA while grounding RoPE, latent projections, score combination, and cache updates.
What you can do after
Build it from scratch as a decoupled MLA attention module with separate content and RoPE-based position streams.
How it connects
Connects Multi-Head Latent Attention, Rotary Positional Encoding, attention scores, the absorption trick, and KV caching.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.