Attention head
An attention head is an independently parameterized attention computation that evaluates relationships among input elements and combines information from them into a contextualized representation.
Natural Language Processing in Action, Second Edition
A self-attention head is an independent attention mechanism that computes relationships among sequence positions within a transformer layer.
- How it is used
- Separates attention into parallel mechanisms so transformers can encode different word relationships and meanings.
- How deep it goes
- Explains query–key scoring, value weighting, head-specific dimensions, concatenation, and output projection.
- Examples
- Uses the original Transformer’s eight heads and BERT base and large configurations as concrete architectural examples.
- What you can do after
- Evaluate transformer configurations by comparing layers, hidden sizes, head counts, and resulting computational dimensions.
Read how Natural Language Processing in Action, Second Edition explains attention head →
Grokking AI Algorithms, Second Edition
An attention head is an independently parameterized component of an attention layer that learns relationships among tokens from a particular perspective using its own query, key, and value matrices.
- How it is used
- Uses parallel perspectives within Transformer self-attention to learn different relationships among tokens.
- How deep it goes
- Explains specialization, independent QKV matrices, output combination, and width allocation, without deriving the full attention equations.
- Examples
- Uses the Alice sentence, chef analogy, grammatical specializations, and alternative head-width configurations.
- What you can do after
- Understand how multiple independently parameterized heads capture and combine different token relationships.
Read how Grokking AI Algorithms, Second Edition explains attention head →
Related concepts
- multi-head attentionpart of · Multi-head attention is an attention mechanism that computes multiple attention operations in parallel using distinct learned projections and combines their outputs.
- attentionuses · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
- grouped-query attentionused by · Grouped-query attention partitions query heads into groups whose members share key and value projections, reducing attention computation and memory use.
- queryproduces · An attention query is a vector representation of the information sought by an input position, compared with key representations to determine which information to attend to.
- matrix-matrix multiplicationuses · Matrix multiplication combines compatible matrices to form a result whose entries are sums of products between rows of the first matrix and columns of the second.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.