Attention head
An attention head is an independently parameterized attention computation that uses queries, keys, and values to weight and combine information across input elements.
CUDA for LLMs
An attention head is an individual attention computation that projects inputs to queries, keys, and values and produces an output.
- How it is used
- Uses attention heads to gather information across token positions in a transformer inference baseline.
- How deep it goes
- Walks through PyTorch head initialization and the masked forward pass, then combines head outputs in multihead attention.
- Examples
- Uses a 64-token transformer with 128-dimensional embeddings, four heads, and eight layers.
- What you can do after
- Apply it using PyTorch to implement a masked attention head and combine multiple heads.
Rearchitecting LLMs
An attention head is an independent attention mechanism that learns to focus on particular types of relationships among sequence elements.
- How it is used
- Enriches each token embedding with sequence-wide context so its representation can reflect neighboring tokens.
- How deep it goes
- Traces parallel heads from smaller projections through contextual analysis and concatenation, while noting that learned specializations are not directly interpretable.
- Examples
- Uses “spring” in contrasting contexts and DistilGPT2’s 12 heads of 64 dimensions each.
- What you can do after
- Understand how parallel heads add contextual signals and inspect a model configuration for head count and head dimension.
Timeless Algorithms, The Foundational Ideas
A component of an attention architecture that evaluates relationships among sequence elements, potentially focusing on a different type of relationship from other heads.
- How it is used
- Uses multiple heads to construct different views of token context and represent several relationship types simultaneously.
- How deep it goes
- Defines heads as distinct views of context, without detailing head-specific computations or parameters.
- Examples
- Uses the trophy–box–it pronoun example to illustrate attention’s contextual relationship weighting.
- What you can do after
- Understand how multiple attention heads represent several kinds of contextual relationships at once.
Read how Timeless Algorithms, The Foundational Ideas explains attention head →
Natural Language Processing in Action, Second Edition
A self-attention head is an independent attention mechanism that computes relationships among sequence positions within a transformer layer.
- How it is used
- Separates attention into parallel mechanisms so transformers can encode different word relationships and meanings.
- How deep it goes
- Explains query–key scoring, value weighting, head-specific dimensions, concatenation, and output projection.
- Examples
- Uses the original Transformer’s eight heads and BERT base and large configurations as concrete architectural examples.
- What you can do after
- Evaluate transformer configurations by comparing layers, hidden sizes, head counts, and resulting computational dimensions.
Read how Natural Language Processing in Action, Second Edition explains attention head →
Grokking AI Algorithms, Second Edition
An attention head is an independently parameterized component of an attention layer that learns relationships among tokens from a particular perspective using its own query, key, and value matrices.
- How it is used
- Uses parallel perspectives within Transformer self-attention to learn different relationships among tokens.
- How deep it goes
- Explains specialization, independent QKV matrices, output combination, and width allocation, without deriving the full attention equations.
- Examples
- Uses the Alice sentence, chef analogy, grammatical specializations, and alternative head-width configurations.
- What you can do after
- Understand how multiple independently parameterized heads capture and combine different token relationships.
Read how Grokking AI Algorithms, Second Edition explains attention head →
Machines that Think
An attention head is a complete attention computation that generates queries, keys, and values, scores token relationships, and combines values.
- How it is used
- Explains how token representations gather context within a large language model.
- How deep it goes
- Traces learned projections, attention weights, and value combination through parallel heads, without formulas or implementation details.
- Examples
- Uses “The sun rises in the east” to illustrate differing attention to a subject and location.
- What you can do after
- Understand how a head combines contextual information and why multiple heads can represent different relationships.
Build a DeepSeek Model (From Scratch)
An attention head is an independently parameterized attention submodule that computes relationship-specific attention within a multi-head attention mechanism.
- How it is used
- Explains how multi-head attention captures several relationship types while motivating the memory cost of per-head key-value caching.
- How deep it goes
- Describes independent projections, specialization, ambiguity resolution, and cache implications, but does not implement tensor operations.
- Examples
- Uses an ambiguous artist sentence, four-head projection diagrams, and per-head key-value cache calculations.
- What you can do after
- Evaluate how independent attention heads trade expressive relationship modeling against key-value cache memory.
Read how Build a DeepSeek Model (From Scratch) explains attention head →
Also covered in
- Build a Text-to-Image Generator (from Scratch)
- Crack Any Codebase with AIIn early access (MEAP)
- Fight Fraud with Machine LearningIn early access (MEAP)
What to know first
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- linear layerA linear layer is a neural-network layer that applies a learned affine transformation to its input, typically using weights and a bias.
- softmaxSoftmax is a function that exponentiates each value in a score vector and normalizes the results to produce positive values summing to one, forming a probability distribution.
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
Related concepts
- multi-head attentionpart of · Multi-head attention is an attention mechanism that computes multiple attention operations in parallel using distinct learned projections and combines their outputs.
- matrix-matrix multiplicationuses · Matrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
- scaled-dot product attentionhas part · Scaled dot-product attention computes query–key dot products, scales them by the square root of the key dimension, applies softmax to obtain weights, and uses those weights to combine value vectors.
- attentionuses · Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- grouped-query attentionused by · Grouped-query attention is an attention mechanism in which multiple query heads share key and value heads or projections.
Questions about attention head
What is attention head?
An attention head is an independently parameterized attention computation that uses queries, keys, and values to weight and combine information across input elements.
Which Manning books explain attention head?
CUDA for LLMs, Rearchitecting LLMs, Timeless Algorithms, The Foundational Ideas, Natural Language Processing in Action, Second Edition, Grokking AI Algorithms, Second Edition, Machines that Think and Build a DeepSeek Model (From Scratch).
What should I know before learning attention head?
Manning books that teach attention head usually introduce transformer architecture, linear layer, softmax and attention first.
What kinds of attention head are there?
Kinds of attention head include evaluator head.
What is attention head part of?
Attention head is part of multi-head attention. Multi-head attention is an attention mechanism that computes multiple attention operations in parallel using distinct learned projections and combines their outputs.
What are the parts of attention head?
Attention head includes scaled-dot product attention.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.






