liveBook · Manning

Attention head

As explained in Build a DeepSeek Model (From Scratch)

An attention head is an independently parameterized attention submodule that computes relationship-specific attention within a multi-head attention mechanism.

An attention head is an independently parameterized submodule within multi-head attention. Each head projects input embeddings through its own query, key, and value weights, then learns to attend to relationships in a distinct representational space. section 2.6.1

The implementation keeps each head's key and value projections separate: with four heads, the layer uses Wk1–Wk4 and corresponding value projections. The resulting K1, K2, and V1, V2 can therefore encode different patterns, while each query asks a different relationship-specific question. section 2.6.1

Attention-head independence provides:

  • Specialization across syntactic, semantic, or descriptive relationships. section 2.6.3
  • Parallel processing of multiple interpretations through different learned projections. section 2.6.3
  • A larger key-value cache, because inference stores separate key and value matrices for every head. section 2.5.1 section 2.6.1

Resolving an ambiguous relationship

In “The artist painted the portrait of a woman with a brush,” one head might connect “painted” with “brush” as an instrument, while another connects “woman” with “brush” as an attribute. Combining their context vectors preserves both interpretations. section 2.6.3

Figure 2.23 Standard Multi-Head Attention (MHA). Each of the four attention heads has its own distinct Key and Value weight matrices, indicated by the different colors. This allows each head to specialize and learn different patterns.

Use the diagram to compare four attention heads with distinct key and value weight matrices, the architectural basis for head specialization. section 2.6.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Describes independent projections, specialization, ambiguity resolution, and cache implications, but does not implement tensor operations.
Examples
Uses an ambiguous artist sentence, four-head projection diagrams, and per-head key-value cache calculations.
What it assumes
Assumes familiarity with transformer layers, queries, keys, values, and attention.
What you can do after
Evaluate how independent attention heads trade expressive relationship modeling against key-value cache memory.
How it connects
Connects attention heads to multi-head attention, multi-query attention, transformers, and the key-value cache.

Also explained in

Compare how 7 books explain attention head →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.