liveBook · Manning

Grouped-query attention

Also called GQA

Grouped-query attention is an attention mechanism in which multiple query heads share key and value heads or projections.

How 3 Manning books explain it

Rearchitecting LLMs

Grouped query attention is an attention mechanism in which multiple query heads share key and value heads.

How it is used
Targets KV-cache memory growth during token generation and informs attention-architecture choices.
How deep it goes
Explains head sharing, cache arithmetic, and projection dimensions, stopping short of implementing a conversion.
Examples
Compares a six-layer cache calculation and inspects Llama-3.2-1B attention projections.
What you can do after
Evaluate attention memory trade-offs and identify GQA from projection dimensions.

Read how Rearchitecting LLMs explains grouped-query attention →

Build a DeepSeek Model (From Scratch)

Grouped-query attention partitions query heads into groups whose members share key and value projections while different groups retain distinct projections.

How it is used
Addresses KV cache memory pressure during autoregressive inference by sharing key and value projections across groups of query heads.
How deep it goes
Develops the cache-size formula and a PyTorch layer with configurable groups, then compares attention variants experimentally.
Examples
Uses Llama 3 8B’s 32 heads and eight groups and a controlled comparison of attention variants.
What you can do after
Build a configurable GQA layer from scratch in PyTorch and evaluate its cache-memory trade-off.

Read how Build a DeepSeek Model (From Scratch) explains grouped-query attention →

Domain-Specific Small Language Models

An attention variant that speeds inference by sharing attention computation across groups of queries.

How it is used
Speeds inference and reduces computational overhead in StarCoder2 for code generation.
How deep it goes
States its inference benefits but does not detail the attention computation.
Examples
Uses a GPU benchmark comparing StarCoder2 throughput and latency with CodeGen 350M and 2B mono models.
What you can do after
Evaluate inference performance by comparing StarCoder2 throughput and latency against CodeGen models on a GPU.

Read how Domain-Specific Small Language Models explains grouped-query attention →

Also covered in

What to know first

Related concepts

Questions about grouped-query attention

What is grouped-query attention?

Grouped-query attention is an attention mechanism in which multiple query heads share key and value heads or projections.

Which Manning books explain grouped-query attention?

Rearchitecting LLMs, Build a DeepSeek Model (From Scratch) and Domain-Specific Small Language Models.

What should I know before learning grouped-query attention?

Manning books that teach grouped-query attention usually introduce context window, Llama, memory usage and parameter first.

What is grouped-query attention also called?

Grouped-query attention is also called GQA.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.