liveBook · Manning

Attention head

An attention head is an independently parameterized attention computation that uses queries, keys, and values to weight and combine information across input elements.

How 7 Manning books explain it

CUDA for LLMs

An attention head is an individual attention computation that projects inputs to queries, keys, and values and produces an output.

How it is used
Uses attention heads to gather information across token positions in a transformer inference baseline.
How deep it goes
Walks through PyTorch head initialization and the masked forward pass, then combines head outputs in multihead attention.
Examples
Uses a 64-token transformer with 128-dimensional embeddings, four heads, and eight layers.
What you can do after
Apply it using PyTorch to implement a masked attention head and combine multiple heads.

Read how CUDA for LLMs explains attention head →

Rearchitecting LLMs

An attention head is an independent attention mechanism that learns to focus on particular types of relationships among sequence elements.

How it is used
Enriches each token embedding with sequence-wide context so its representation can reflect neighboring tokens.
How deep it goes
Traces parallel heads from smaller projections through contextual analysis and concatenation, while noting that learned specializations are not directly interpretable.
Examples
Uses “spring” in contrasting contexts and DistilGPT2’s 12 heads of 64 dimensions each.
What you can do after
Understand how parallel heads add contextual signals and inspect a model configuration for head count and head dimension.

Read how Rearchitecting LLMs explains attention head →

Timeless Algorithms, The Foundational Ideas

A component of an attention architecture that evaluates relationships among sequence elements, potentially focusing on a different type of relationship from other heads.

How it is used
Uses multiple heads to construct different views of token context and represent several relationship types simultaneously.
How deep it goes
Defines heads as distinct views of context, without detailing head-specific computations or parameters.
Examples
Uses the trophy–box–it pronoun example to illustrate attention’s contextual relationship weighting.
What you can do after
Understand how multiple attention heads represent several kinds of contextual relationships at once.

Read how Timeless Algorithms, The Foundational Ideas explains attention head →

Natural Language Processing in Action, Second Edition

A self-attention head is an independent attention mechanism that computes relationships among sequence positions within a transformer layer.

How it is used
Separates attention into parallel mechanisms so transformers can encode different word relationships and meanings.
How deep it goes
Explains query–key scoring, value weighting, head-specific dimensions, concatenation, and output projection.
Examples
Uses the original Transformer’s eight heads and BERT base and large configurations as concrete architectural examples.
What you can do after
Evaluate transformer configurations by comparing layers, hidden sizes, head counts, and resulting computational dimensions.

Read how Natural Language Processing in Action, Second Edition explains attention head →

Grokking AI Algorithms, Second Edition

An attention head is an independently parameterized component of an attention layer that learns relationships among tokens from a particular perspective using its own query, key, and value matrices.

How it is used
Uses parallel perspectives within Transformer self-attention to learn different relationships among tokens.
How deep it goes
Explains specialization, independent QKV matrices, output combination, and width allocation, without deriving the full attention equations.
Examples
Uses the Alice sentence, chef analogy, grammatical specializations, and alternative head-width configurations.
What you can do after
Understand how multiple independently parameterized heads capture and combine different token relationships.

Read how Grokking AI Algorithms, Second Edition explains attention head →

Machines that Think

An attention head is a complete attention computation that generates queries, keys, and values, scores token relationships, and combines values.

How it is used
Explains how token representations gather context within a large language model.
How deep it goes
Traces learned projections, attention weights, and value combination through parallel heads, without formulas or implementation details.
Examples
Uses “The sun rises in the east” to illustrate differing attention to a subject and location.
What you can do after
Understand how a head combines contextual information and why multiple heads can represent different relationships.

Read how Machines that Think explains attention head →

Build a DeepSeek Model (From Scratch)

An attention head is an independently parameterized attention submodule that computes relationship-specific attention within a multi-head attention mechanism.

How it is used
Explains how multi-head attention captures several relationship types while motivating the memory cost of per-head key-value caching.
How deep it goes
Describes independent projections, specialization, ambiguity resolution, and cache implications, but does not implement tensor operations.
Examples
Uses an ambiguous artist sentence, four-head projection diagrams, and per-head key-value cache calculations.
What you can do after
Evaluate how independent attention heads trade expressive relationship modeling against key-value cache memory.

Read how Build a DeepSeek Model (From Scratch) explains attention head →

Also covered in

What to know first

Related concepts

Questions about attention head

What is attention head?

An attention head is an independently parameterized attention computation that uses queries, keys, and values to weight and combine information across input elements.

Which Manning books explain attention head?

CUDA for LLMs, Rearchitecting LLMs, Timeless Algorithms, The Foundational Ideas, Natural Language Processing in Action, Second Edition, Grokking AI Algorithms, Second Edition, Machines that Think and Build a DeepSeek Model (From Scratch).

What should I know before learning attention head?

Manning books that teach attention head usually introduce transformer architecture, linear layer, softmax and attention first.

What kinds of attention head are there?

Kinds of attention head include evaluator head.

What is attention head part of?

Attention head is part of multi-head attention. Multi-head attention is an attention mechanism that computes multiple attention operations in parallel using distinct learned projections and combines their outputs.

What are the parts of attention head?

Attention head includes scaled-dot product attention.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.