liveBook · Manning

Latent space

As explained in Build a DeepSeek Model (From Scratch)

A latent space is a lower-level representational space in which compressed or abstract features encode information from the original representations.

A latent space is a smaller, lower-dimensional representational space that compresses information before it is reconstructed for computation. In Multi-Head Latent Attention (MLA), the key/value path projects each input into a compact latent matrix, caches that matrix, and reconstructs full Keys and Values when needed. section 3.1 section 3.2.2

The MLA latent-space workflow is:

  • Down-project the input embedding matrix X with the KV Down-Projector Wdkv to produce the compressed latent KV matrix cKV. section 3.2.2
  • Cache one compact cKV matrix per token instead of separately caching full-dimensional Keys and Values. section 3.1 section 3.2.2
  • Up-project cKV through separate Key and Value matrices, Wuk and Wuv, to reconstruct full K and V for attention. section 3.2.2
  • Use associative matrix rearrangement to reuse the latent matrix in the final context calculation, while absorbing Wuv into the output projection when appropriate. section 3.3.3

From 7168 dimensions to 512

For a DeepSeek-V3-sized example, Wdkv maps a 7168-dimensional model vector to a 512-dimensional latent vector. The simplified diagram uses X with shape (4, 8), Wdkv with shape (8, 4), and cKV with shape (4, 4); those small dimensions illustrate the mechanism rather than model proportions. section 3.2.2

Figure 3.2 The full architectural data flow of Multi-Head Latent Attention (MLA).

Use the architectural data-flow figure to trace the query path alongside the compressed key/value path. section 3.2

Listing 3.1 Building MLA From Scratch
 
import torch
import torch.nn as nn
 
class MultiHeadLatentAttention(nn.Module):
    def __init__(self, d_model, num_heads, d_latent, 
    [CA] dropout=0.0):
        super().__init__()
        assert d_model % num_heads == 0, \
            "d_model must be divisible by num_heads"
 
        self.d_model = d_model
        self.num_heads = num_heads
        self.d_head = d_model // num_heads
        self.d_latent = d_latent
 
        self.W_q = nn.Linear(d_model, d_model)  #A
        self.W_dkv = nn.Linear(d_model, d_latent)  #B
 
        self.W_uk = nn.Linear(d_latent, d_model) #C
        self.W_uv = nn.Linear(d_latent, d_model) #C
 
        self.W_o = nn.Linear(d_model, d_model)
        self.dropout = nn.Dropout(dropout)
        self.register_buffer('mask', torch.triu(torch.ones(
        [CA]     1, 1, 1024, 1024), diagonal=1).bool())
 
    def forward(self, x):
        batch_size, seq_len, _ = x.shape
 
        # 1. Query Path
        q = self.W_q(x).view(
        [CA]         batch_size, seq_len, self.num_heads, self.d_head
        [CA]     ).transpose(1, 2)
 
        # 2. Key/Value Path (The MLA Innovation)
        # Down-project to the latent space
        c_kv = self.W_dkv(x)  #D
 
        # Up-project from latent space to get full K and V
        k = self.W_uk(c_kv).view(
        [CA]         batch_size, seq_len, self.num_heads, self.d_head
        [CA]     ).transpose(1, 2)  #E
        v = self.W_uv(c_kv).view(
        [CA]         batch_size, seq_len, self.num_heads, self.d_head
        [CA]     ).transpose(1, 2)  #E
 
        # 3. Standard Attention Calculation
        attn_scores = (q @ k.transpose(-2, -1)) / \
        [CA]         (self.d_head ** 0.5)
 
        attn_scores = attn_scores.masked_fill(
        [CA]         self.mask[:, :, :seq_len, :seq_len], float('-inf'))
 
        attn_weights = torch.softmax(attn_scores, dim=-1)
        attn_weights = self.dropout(attn_weights)
 
        context_vector = (attn_weights @ v).transpose(1, 2) \
        [CA]         .contiguous().view(batch_size, seq_len, self.d_model)
 
        # 4. Final Output Projection
        output = self.W_o(context_vector)
        return output

Use the PyTorch listing to inspect the down-projection, up-projections, output projection, dropout, and causal mask in a complete MLA module. section 3.6

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains projection shapes, matrix rearrangement, and PyTorch construction, while deferring positional encodings and omitting RMSNorm from the simplified module.
Examples
Uses DeepSeek-V3 dimensions, toy matrix shapes, PyTorch projections, and associative formulas for reconstructing Values and context vectors.
What it assumes
Assumes familiarity with standard attention, input embeddings, matrix multiplication, and projection layers before introducing compressed key/value representations.
What you can do after
Build it from scratch in PyTorch with down-projection, up-projection, and attention calculation, while handling caching and positional encoding separately.
How it connects
Connects latent representations with Multi-Head Latent Attention, key/value projections, inference caching, attention calculation, positional encodings, and RMSNorm.

Also explained in

Compare how 22 books explain latent space →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.