A latent space is a smaller, lower-dimensional representational space that compresses information before it is reconstructed for computation. In Multi-Head Latent Attention (MLA), the key/value path projects each input into a compact latent matrix, caches that matrix, and reconstructs full Keys and Values when needed. section 3.1 section 3.2.2
The MLA latent-space workflow is:
- Down-project the input embedding matrix X with the KV Down-Projector Wdkv to produce the compressed latent KV matrix cKV. section 3.2.2
- Cache one compact cKV matrix per token instead of separately caching full-dimensional Keys and Values. section 3.1 section 3.2.2
- Up-project cKV through separate Key and Value matrices, Wuk and Wuv, to reconstruct full K and V for attention. section 3.2.2
- Use associative matrix rearrangement to reuse the latent matrix in the final context calculation, while absorbing Wuv into the output projection when appropriate. section 3.3.3
From 7168 dimensions to 512
For a DeepSeek-V3-sized example, Wdkv maps a 7168-dimensional model vector to a 512-dimensional latent vector. The simplified diagram uses X with shape (4, 8), Wdkv with shape (8, 4), and cKV with shape (4, 4); those small dimensions illustrate the mechanism rather than model proportions. section 3.2.2
Figure 3.2 The full architectural data flow of Multi-Head Latent Attention (MLA).

Use the architectural data-flow figure to trace the query path alongside the compressed key/value path. section 3.2
Listing 3.1 Building MLA From Scratch
import torch
import torch.nn as nn
class MultiHeadLatentAttention(nn.Module):
def __init__(self, d_model, num_heads, d_latent,
[CA] dropout=0.0):
super().__init__()
assert d_model % num_heads == 0, \
"d_model must be divisible by num_heads"
self.d_model = d_model
self.num_heads = num_heads
self.d_head = d_model // num_heads
self.d_latent = d_latent
self.W_q = nn.Linear(d_model, d_model) #A
self.W_dkv = nn.Linear(d_model, d_latent) #B
self.W_uk = nn.Linear(d_latent, d_model) #C
self.W_uv = nn.Linear(d_latent, d_model) #C
self.W_o = nn.Linear(d_model, d_model)
self.dropout = nn.Dropout(dropout)
self.register_buffer('mask', torch.triu(torch.ones(
[CA] 1, 1, 1024, 1024), diagonal=1).bool())
def forward(self, x):
batch_size, seq_len, _ = x.shape
# 1. Query Path
q = self.W_q(x).view(
[CA] batch_size, seq_len, self.num_heads, self.d_head
[CA] ).transpose(1, 2)
# 2. Key/Value Path (The MLA Innovation)
# Down-project to the latent space
c_kv = self.W_dkv(x) #D
# Up-project from latent space to get full K and V
k = self.W_uk(c_kv).view(
[CA] batch_size, seq_len, self.num_heads, self.d_head
[CA] ).transpose(1, 2) #E
v = self.W_uv(c_kv).view(
[CA] batch_size, seq_len, self.num_heads, self.d_head
[CA] ).transpose(1, 2) #E
# 3. Standard Attention Calculation
attn_scores = (q @ k.transpose(-2, -1)) / \
[CA] (self.d_head ** 0.5)
attn_scores = attn_scores.masked_fill(
[CA] self.mask[:, :, :seq_len, :seq_len], float('-inf'))
attn_weights = torch.softmax(attn_scores, dim=-1)
attn_weights = self.dropout(attn_weights)
context_vector = (attn_weights @ v).transpose(1, 2) \
[CA] .contiguous().view(batch_size, seq_len, self.d_model)
# 4. Final Output Projection
output = self.W_o(context_vector)
return outputUse the PyTorch listing to inspect the down-projection, up-projections, output projection, dropout, and causal mask in a complete MLA module. section 3.6





















