liveBook · Manning

Multi-query attention

As explained in Build a DeepSeek Model (From Scratch)

Multi-query attention uses multiple query heads that share a smaller set of key and value projections to reduce attention-memory requirements.

Multi-query attention (MQA) reduces key-value (KV) cache memory during autoregressive inference. Each attention head has its own query projection, but all heads share one key projection and one value projection. In standard multi-head attention (MHA), each head has distinct key and value projections. section 2.5.2 section 2.6.1

Figure 2.24 Multi-Query Attention (MQA). All four heads still have unique Query projections, but they now share a single, common Key and Value projection, indicated by the uniform light blue and yellow colors.

Four query heads share a single key and value projection. section 2.6.1

For n query heads of the same head size, MQA stores one KV pair instead of n pairs, reducing KV cache size by a factor of n relative to MHA. With 16-bit storage, the cache size in bytes is l × b × 1 × h × s × 2 × 2, where l is the number of layers, b the batch size, h the head size, and s the sequence length. The first factor of 2 stores keys and values; the second counts bytes per element. section 2.6.2 section 2.7.2

A counterfactual cache estimate

Applying MQA to GPT-3's 96-head dimensions would reduce the illustrated KV cache from 4.5 GB to 48 MB. GPT-3 does not actually use MQA; this is a comparison using its dimensions. MQA changes the architecture and generally requires retraining or uptraining, rather than simply switching on a cache. section 2.6.2

The savings come with a trade-off: shared keys and values limit the independent representations available to different heads, reducing their ability to specialize. Grouped-query attention (GQA) offers a middle ground: groups of query heads share distinct KV pairs. One group corresponds to MQA; one group per head corresponds to MHA. section 2.6.3 section 2.7.2

To implement MQA, project queries from d_model to d_model and keys and values from d_model to d_head, where d_head = d_model / num_heads. Cache each shared key and value with shape (batch_size, 1, seq_len, d_head), rather than (batch_size, num_heads, seq_len, d_head). The illustrative PyTorch layer repeats keys and values across heads for clarity; .repeat() physically copies data, unlike the view returned by .expand(). section 2.6.4

Listing 2.3 Implementing an MQA layer from scratch
import torch
import torch.nn as nn
 
class MultiQueryAttention(nn.Module):
    def __init__(self, d_model, num_heads, dropout=0.0):
        super().__init__()
        assert d_model % num_heads == 0, \
            "d_model must be divisible by num_heads"
 
        self.d_model = d_model
        self.num_heads = num_heads
        self.d_head = d_model // num_heads
 
        self.W_q = nn.Linear(d_model, d_model)     #A
        self.W_k = nn.Linear(d_model, self.d_head) #B
        self.W_v = nn.Linear(d_model, self.d_head) #B
        self.W_o = nn.Linear(d_model, d_model)
 
        self.dropout = nn.Dropout(dropout)
        self.register_buffer('mask', torch.triu(
        [CA] torch.ones(1, 1, 1024, 1024), diagonal=1))
 
    def forward(self, x):
        batch_size, seq_len, _ = x.shape
 
        q = self.W_q(x).view(batch_size, seq_len, self.num_heads, 
        [CA] self.d_head).transpose(1, 2)
 
        k = self.W_k(x).view(batch_size, seq_len, 1, 
        [CA] self.d_head).transpose(1, 2)
        v = self.W_v(x).view(batch_size, seq_len, 1, 
        [CA] self.d_head).transpose(1, 2)
 
        k = k.repeat(1, self.num_heads, 1, 1)  #C
        v = v.repeat(1, self.num_heads, 1, 1)  #C
 
        attn_scores = (q @ k.transpose(-2, -1)) / (self.d_head ** 0.5)
 
        attn_scores = attn_scores.masked_fill(
        [CA] self.mask[:,:,:seq_len,:seq_len] == 1, float('-inf'))
 
        attn_weights = torch.softmax(attn_scores, dim=-1)
        attn_weights = self.dropout(attn_weights)
 
        context_vector = (attn_weights @ v).transpose(1, 2) \
        [CA] .contiguous().view(batch_size, seq_len, self.d_model)
 
        output = self.W_o(context_vector)
        return output

Inspect the PyTorch module and its query, key, and value projections. section 2.6.4

A controlled comparison

In a four-model experiment, MQA used a 6 MB KV cache and reached a best validation loss of 2.4080; MHA used 48 MB and reached 2.3721. These results illustrate the memory–quality trade-off for that experiment, not a universal performance guarantee. section 3.17

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Derives cache savings, explains the expressivity trade-off, and implements a PyTorch attention layer.
Examples
Uses a counterfactual GPT-3 cache estimate and a controlled four-architecture validation-loss and memory comparison.
What you can do after
Build it from scratch as a PyTorch layer with shared key and value projections.
How it connects
Connects multi-head attention, grouped-query attention, KV caching, and multi-head latent attention through their memory–expressivity trade-offs.

Also explained in

Compare how 2 books explain multi-query attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.