Multi-query attention (MQA) reduces key-value (KV) cache memory during autoregressive inference. Each attention head has its own query projection, but all heads share one key projection and one value projection. In standard multi-head attention (MHA), each head has distinct key and value projections. section 2.5.2 section 2.6.1
Figure 2.24 Multi-Query Attention (MQA). All four heads still have unique Query projections, but they now share a single, common Key and Value projection, indicated by the uniform light blue and yellow colors.

Four query heads share a single key and value projection. section 2.6.1
For n query heads of the same head size, MQA stores one KV pair instead of n pairs, reducing KV cache size by a factor of n relative to MHA. With 16-bit storage, the cache size in bytes is l × b × 1 × h × s × 2 × 2, where l is the number of layers, b the batch size, h the head size, and s the sequence length. The first factor of 2 stores keys and values; the second counts bytes per element. section 2.6.2 section 2.7.2
A counterfactual cache estimate
Applying MQA to GPT-3's 96-head dimensions would reduce the illustrated KV cache from 4.5 GB to 48 MB. GPT-3 does not actually use MQA; this is a comparison using its dimensions. MQA changes the architecture and generally requires retraining or uptraining, rather than simply switching on a cache. section 2.6.2
The savings come with a trade-off: shared keys and values limit the independent representations available to different heads, reducing their ability to specialize. Grouped-query attention (GQA) offers a middle ground: groups of query heads share distinct KV pairs. One group corresponds to MQA; one group per head corresponds to MHA. section 2.6.3 section 2.7.2
To implement MQA, project queries from d_model to d_model and keys and values from d_model to d_head, where d_head = d_model / num_heads. Cache each shared key and value with shape (batch_size, 1, seq_len, d_head), rather than (batch_size, num_heads, seq_len, d_head). The illustrative PyTorch layer repeats keys and values across heads for clarity; .repeat() physically copies data, unlike the view returned by .expand(). section 2.6.4
Listing 2.3 Implementing an MQA layer from scratch
import torch
import torch.nn as nn
class MultiQueryAttention(nn.Module):
def __init__(self, d_model, num_heads, dropout=0.0):
super().__init__()
assert d_model % num_heads == 0, \
"d_model must be divisible by num_heads"
self.d_model = d_model
self.num_heads = num_heads
self.d_head = d_model // num_heads
self.W_q = nn.Linear(d_model, d_model) #A
self.W_k = nn.Linear(d_model, self.d_head) #B
self.W_v = nn.Linear(d_model, self.d_head) #B
self.W_o = nn.Linear(d_model, d_model)
self.dropout = nn.Dropout(dropout)
self.register_buffer('mask', torch.triu(
[CA] torch.ones(1, 1, 1024, 1024), diagonal=1))
def forward(self, x):
batch_size, seq_len, _ = x.shape
q = self.W_q(x).view(batch_size, seq_len, self.num_heads,
[CA] self.d_head).transpose(1, 2)
k = self.W_k(x).view(batch_size, seq_len, 1,
[CA] self.d_head).transpose(1, 2)
v = self.W_v(x).view(batch_size, seq_len, 1,
[CA] self.d_head).transpose(1, 2)
k = k.repeat(1, self.num_heads, 1, 1) #C
v = v.repeat(1, self.num_heads, 1, 1) #C
attn_scores = (q @ k.transpose(-2, -1)) / (self.d_head ** 0.5)
attn_scores = attn_scores.masked_fill(
[CA] self.mask[:,:,:seq_len,:seq_len] == 1, float('-inf'))
attn_weights = torch.softmax(attn_scores, dim=-1)
attn_weights = self.dropout(attn_weights)
context_vector = (attn_weights @ v).transpose(1, 2) \
[CA] .contiguous().view(batch_size, seq_len, self.d_model)
output = self.W_o(context_vector)
return outputInspect the PyTorch module and its query, key, and value projections. section 2.6.4
A controlled comparison
In a four-model experiment, MQA used a 6 MB KV cache and reached a best validation loss of 2.4080; MHA used 48 MB and reached 2.3721. These results illustrate the memory–quality trade-off for that experiment, not a universal performance guarantee. section 3.17

