An attention head evaluates relationships among sequence elements and contributes to contextual representations. Multiple heads can form different views of the same context, representing several kinds of relationships at once. This distributes context across token representations rather than relying solely on one recurrent state, but finite vector dimensions and context-window limits remain. section 12.3.2 section 12.2.1
Attention head
A component of an attention architecture that evaluates relationships among sequence elements, potentially focusing on a different type of relationship from other heads.
How Timeless Algorithms, The Foundational Ideas covers it
- How deep it goes
- Defines heads as distinct views of context, without detailing head-specific computations or parameters.
- Examples
- Uses the trophy–box–it pronoun example to illustrate attention’s contextual relationship weighting.
- What you can do after
- Understand how multiple attention heads represent several kinds of contextual relationships at once.
- How it connects
- Connects attention heads to self-attention, token representations, recurrent state, and context windows.
Also explained in
-
CUDA for LLMsAn attention head is an individual attention computation that projects inputs to queries, keys, and values and produces an output.
-
Rearchitecting LLMsAn attention head is an independent attention mechanism that learns to focus on particular types of relationships among sequence elements.
-
Natural Language Processing in Action, Second EditionA self-attention head is an independent attention mechanism that computes relationships among sequence positions within a transformer layer.
-
Grokking AI Algorithms, Second EditionAn attention head is an independently parameterized component of an attention layer that learns relationships among tokens from a particular perspective using its own query, key, and value matrices.
-
Machines that ThinkAn attention head is a complete attention computation that generates queries, keys, and values, scores token relationships, and combines values.
-
Build a DeepSeek Model (From Scratch)An attention head is an independently parameterized attention submodule that computes relationship-specific attention within a multi-head attention mechanism.
Compare how 7 books explain attention head →
Related concepts
- multi-head attentionpart of · Multi-head attention is an attention mechanism that computes multiple attention operations in parallel using distinct learned projections and combines their outputs.
- matrix-matrix multiplicationuses · Matrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
- scaled-dot product attentionhas part · Scaled dot-product attention computes query–key dot products, scales them by the square root of the key dimension, applies softmax to obtain weights, and uses those weights to combine value vectors.
- attentionuses · Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- grouped-query attentionused by · Grouped-query attention is an attention mechanism in which multiple query heads share key and value heads or projections.
This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.






