liveBook · Manning

Attention head

As explained in Rearchitecting LLMs

An attention head is an independent attention mechanism that learns to focus on particular types of relationships among sequence elements.

An attention head is an independent attention mechanism that enriches each token embedding with information from the rest of the sequence. section 3.1.2

In multi-head attention:

  • Heads process embeddings in parallel, each focusing on different relationships learned during training. section 3.1.2
  • Their outputs are concatenated to produce context-aware embeddings; specialization labels are illustrative, not directly interpretable. section 3.1.2

Context and dimensions

The token “spring” can be contextualized differently by “water” and “mountain” or by “flowers” and “season.” DistilGPT2 has 12 heads, each projecting a 768-dimensional embedding into 64 dimensions. section 3.1.2

Figure 3.3 Multi-Head Attention architecture. The input embedding (768 dimensions) is divided into 12 specialized attention heads, each processing 64 dimensions in parallel. Each head focuses on different contextual relationships, before concatenating their outputs into a context-aware embedding that incorporates information from the entire sequence. Each head focuses on different contextual relationships (labels are illustrative; real specialization emerges from training and is not directly interpretable).

Follow the parallel head paths from the input embedding through concatenation into a context-aware embedding. section 3.1.2

How Rearchitecting LLMs covers it

How deep it goes
Traces parallel heads from smaller projections through contextual analysis and concatenation, while noting that learned specializations are not directly interpretable.
Examples
Uses “spring” in contrasting contexts and DistilGPT2’s 12 heads of 64 dimensions each.
What it assumes
Assumes token and positional embeddings, then explains how attention adds information from neighboring sequence elements.
What you can do after
Understand how parallel heads add contextual signals and inspect a model configuration for head count and head dimension.
How it connects
Connects attention heads to token embeddings, positional embeddings, Query-Key-Value attention, multi-head attention, and grouped-query attention.

Also explained in

Compare how 7 books explain attention head →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.