liveBook · Manning

Grouped-query attention

As explained in Rearchitecting LLMs

Grouped query attention is an attention mechanism in which multiple query heads share key and value heads.

Grouped-query attention (GQA) reduces the key-value (KV) cache needed during generation. Each query head has its own query, but heads in a group share key and value heads. The cache still grows with each processed token, but fewer KV heads mean fewer cached vectors per token. section 3.1.2 section 3.2.1

In multi-head attention (MHA), each query head has its own key-value pair. At the other extreme, multi-query attention (MQA) shares one pair across all query heads. GQA lies between them, capturing much of MQA’s memory savings while avoiding its quality compromises. section 3.1.2

Figure 3.5 Attention mechanisms and memory optimization strategies. Multi-Head Attention (MHA) maintains independent K,V pairs for each Q head — one pair per processed token, forming a growing cache. Grouped-Query Attention (GQA) groups Q heads to share K,V pairs (here 6 Q heads in 3 groups, each sharing one K,V pair per token). Multi-Query Attention (MQA) compresses to the extreme: all Q heads share a single K,V pair per token.

Compare the head arrangements: six query heads share three KV pairs in the GQA example. section 3.1.2

Cache budget at a 4:1 grouping ratio

Consider six layers and 12 query heads. MHA caches 8 tokens × 12 KV heads × 2 vectors × 6 layers = 1,152 vectors. With three KV heads, GQA caches 32 tokens × 3 KV heads × 2 vectors × 6 layers = 1,152 vectors. This is an illustrative vector count, not a measured byte-level benchmark. section 3.2.1 section 3.1.2

Projection dimensions reveal the grouping in Llama-3.2-1B: the query projection outputs 2,048 features, while the key and value projections each output 512. The output projection recombines attention-head outputs into an embedding. The model uses a 4:1 query-to-KV grouping ratio. section 3.2.1 section 3.3

(q_proj): Linear(in_features=2048, out_features=2048, 
                             ➥ bias=False)  #A
(k_proj): Linear(in_features=2048, out_features=512, bias=False)   
(v_proj): Linear(in_features=2048, out_features=512, bias=False)   
(o_proj): Linear(in_features=2048, out_features=2048, bias=False)  #B

Inspect the separate query, key, value, and output projection dimensions. section 3.2.1

Shared KV projections also complicate attention-head pruning: unlike in classic MHA, heads in the same group are not independent across all three projections. section 3.2.1

How Rearchitecting LLMs covers it

How deep it goes
Explains head sharing, cache arithmetic, and projection dimensions, stopping short of implementing a conversion.
Examples
Compares a six-layer cache calculation and inspects Llama-3.2-1B attention projections.
What you can do after
Evaluate attention memory trade-offs and identify GQA from projection dimensions.
How it connects
Connects multi-head attention, multi-query attention, KV caching, and attention-head pruning.

Also explained in

Compare how 3 books explain grouped-query attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.