Grouped-query attention
Grouped-query attention is an attention mechanism in which multiple query heads share key and value heads or projections.
Rearchitecting LLMs
Grouped query attention is an attention mechanism in which multiple query heads share key and value heads.
- How it is used
- Targets KV-cache memory growth during token generation and informs attention-architecture choices.
- How deep it goes
- Explains head sharing, cache arithmetic, and projection dimensions, stopping short of implementing a conversion.
- Examples
- Compares a six-layer cache calculation and inspects Llama-3.2-1B attention projections.
- What you can do after
- Evaluate attention memory trade-offs and identify GQA from projection dimensions.
Read how Rearchitecting LLMs explains grouped-query attention →
Build a DeepSeek Model (From Scratch)
Grouped-query attention partitions query heads into groups whose members share key and value projections while different groups retain distinct projections.
- How it is used
- Addresses KV cache memory pressure during autoregressive inference by sharing key and value projections across groups of query heads.
- How deep it goes
- Develops the cache-size formula and a PyTorch layer with configurable groups, then compares attention variants experimentally.
- Examples
- Uses Llama 3 8B’s 32 heads and eight groups and a controlled comparison of attention variants.
- What you can do after
- Build a configurable GQA layer from scratch in PyTorch and evaluate its cache-memory trade-off.
Read how Build a DeepSeek Model (From Scratch) explains grouped-query attention →
Domain-Specific Small Language Models
An attention variant that speeds inference by sharing attention computation across groups of queries.
- How it is used
- Speeds inference and reduces computational overhead in StarCoder2 for code generation.
- How deep it goes
- States its inference benefits but does not detail the attention computation.
- Examples
- Uses a GPU benchmark comparing StarCoder2 throughput and latency with CodeGen 350M and 2B mono models.
- What you can do after
- Evaluate inference performance by comparing StarCoder2 throughput and latency against CodeGen models on a GPU.
Read how Domain-Specific Small Language Models explains grouped-query attention →
Also covered in
- Quantization and Fast InferenceIn early access (MEAP)
What to know first
- context windowA context window is a bounded span of sequence elements or contextual information that a model or processing operation can consider together.
- LlamaLlama is a family of large language models developed by Meta.
- memory usageMemory usage is the amount of computer memory occupied, allocated, or required by a program, system, data structure, or computation during operation.
- parameterA parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
Related concepts
- multi-head attentionequivalent to · Multi-head attention is an attention mechanism that computes multiple attention operations in parallel using distinct learned projections and combines their outputs.
- multi-query attentionequivalent to · Multi-query attention is an attention architecture in which multiple query heads share key and value projections.
- key-value cacheaddresses · A key-value cache stores attention keys and values computed for earlier tokens so they can be reused during subsequent autoregressive generation.
- keyuses · In an attention mechanism, a key is a vector representation of an input element that is compared with a query to determine the relevance of its associated value.
- attentionuses · Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- attention headuses · An attention head is an independently parameterized attention computation that uses queries, keys, and values to weight and combine information across input elements.
Questions about grouped-query attention
What is grouped-query attention?
Grouped-query attention is an attention mechanism in which multiple query heads share key and value heads or projections.
Which Manning books explain grouped-query attention?
Rearchitecting LLMs, Build a DeepSeek Model (From Scratch) and Domain-Specific Small Language Models.
What should I know before learning grouped-query attention?
Manning books that teach grouped-query attention usually introduce context window, Llama, memory usage and parameter first.
What is grouped-query attention also called?
Grouped-query attention is also called GQA.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


