Multi-query attention
Multi-query attention is an attention architecture in which multiple query heads share key and value projections.
Build a DeepSeek Model (From Scratch)
Multi-query attention uses multiple query heads that share a smaller set of key and value projections to reduce attention-memory requirements.
- How it is used
- Reduces KV cache memory during autoregressive inference by sharing key and value projections across query heads.
- How deep it goes
- Derives cache savings, explains the expressivity trade-off, and implements a PyTorch attention layer.
- Examples
- Uses a counterfactual GPT-3 cache estimate and a controlled four-architecture validation-loss and memory comparison.
- What you can do after
- Build it from scratch as a PyTorch layer with shared key and value projections.
Read how Build a DeepSeek Model (From Scratch) explains multi-query attention →
Rearchitecting LLMs
Multi-query attention is an attention architecture in which multiple query heads share a single set of key and value vectors.
- How it is used
- Reduces key-value cache memory during token-by-token generation by sharing one key-value set across query heads.
- How deep it goes
- Explains shared key-value storage and illustrates cache-memory trade-offs against multi-head and grouped-query attention.
- Examples
- Compares cache capacity for a six-layer, 12-head example using a 1,152-vector budget.
- What you can do after
- Evaluate MQA's cache-memory advantage against its stated quality compromise relative to grouped-query attention.
Read how Rearchitecting LLMs explains multi-query attention →
Related concepts
- grouped-query attentionequivalent to · Grouped-query attention is an attention mechanism in which multiple query heads share key and value heads or projections.
- key-value cacheconfigures · A key-value cache stores attention keys and values computed for earlier tokens so they can be reused during subsequent autoregressive generation.
- multi-head attentioncontrasts with · Multi-head attention is an attention mechanism that computes multiple attention operations in parallel using distinct learned projections and combines their outputs.
- attentionkind of · Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- expressive powercontrasts with · Expressive power is the capacity of a model, language, or system to represent or convey a range and complexity of functions, patterns, distinctions, or ideas.
Questions about multi-query attention
What is multi-query attention?
Multi-query attention is an attention architecture in which multiple query heads share key and value projections.
Which Manning books explain multi-query attention?
Build a DeepSeek Model (From Scratch) and Rearchitecting LLMs.
What is the difference between multi-query attention and multi-head attention?
Multi-query attention is an attention architecture in which multiple query heads share key and value projections. Multi-head attention is an attention mechanism that computes multiple attention operations in parallel using distinct learned projections and combines their outputs.
What is multi-query attention a kind of?
Multi-query attention is a kind of attention. Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

