liveBook · Manning

Multi-head attention

As explained in Building Specialized AI Systems

An attention method that runs attention across multiple learned representation subspaces and concatenates the resulting outputs.

Multi-head attention runs attention in parallel across several learned representation subspaces, then concatenates the outputs. Each head can focus on a different aspect of context, giving every token a richer context-aware representation. section 3.1.1

The underlying attention operation follows a compact pipeline:

  • Project token embeddings into queries, keys, and values. section 3.1.1
  • Compute scaled dot-product relevance scores between positions. section 3.1.1
  • Apply softmax to produce attention weights and form weighted sums of values. section 3.1.1
  • Concatenate the parallel head outputs before subsequent Transformer processing. section 3.1.1

Resolving context

In “The cat sat on the mat because it was tired,” self-attention can connect “it” with “the cat.” Multiple heads allow the model to attend to different contextual relationships simultaneously. section 3.1.1

How Building Specialized AI Systems covers it

How deep it goes
Explains QKV projections, scaled scores, softmax weights, value aggregation, and head concatenation, but stops short of implementation code.
Examples
Uses pronoun resolution, scaled dot-product attention, and an encoder-decoder Transformer architecture as concrete teaching examples.
What it assumes
Builds from token embeddings, queries, keys, values, dot products, softmax, and weighted sums rather than assuming them fully understood.
What you can do after
Evaluate how parallel attention heads create context-aware representations within Transformer encoder and decoder blocks.
How it connects
Connects attention with self-attention, cross-attention, causal masking, Transformer encoders, Transformer decoders, residual connections, and layer normalization.

Also explained in

Compare how 25 books explain multi-head attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.