Attention head

As explained in Natural Language Processing in Action, Second Edition

A self-attention head is an independent attention mechanism that computes relationships among sequence positions within a transformer layer.

An attention head is one independent self-attention mechanism inside a transformer layer. Multiple heads attend to different aspects of the same sequence, allowing distinct word relationships or meanings to be represented in separate vector subspaces. section 9.1.2

Each head projects input embeddings into query, key, and value matrices. Query–key multiplication produces a square connection or scoring matrix; normalized scores weight the values. The heads’ outputs are concatenated and projected through an output matrix. section 9.1.2

A multi-head attention computation proceeds by:

• Creating separate query, key, and value projections with head-specific dimensions. section 9.1.2
• Computing attention scores that connect positions across the input sequence. section 9.1.2
• Concatenating the value outputs from all heads and applying the shared output projection. section 9.1.2

Transformer and BERT configurations

The original Transformer uses eight attention heads, with key and value dimensions of 64 when the model dimension is divided evenly. BERT base uses 12 layers, hidden size 768, and 12 heads; BERT large uses 24 layers, hidden size 1,024, and 16 heads. section 9.1.2 section 9.3.3

Equation 9.2 Multi-head self-attention

Use the multi-head self-attention equation to inspect how head-specific computations combine into one transformer output. section 9.1.2

How Natural Language Processing in Action, Second Edition covers it

How deep it goes

Explains query–key scoring, value weighting, head-specific dimensions, concatenation, and output projection.

Examples

Uses the original Transformer’s eight heads and BERT base and large configurations as concrete architectural examples.

What it assumes

Builds on token embeddings and self-attention while contrasting the mechanism with recurrent neural networks.

What you can do after

Evaluate transformer configurations by comparing layers, hidden sizes, head counts, and resulting computational dimensions.

How it connects

Connects attention heads with self-attention, embeddings, transformer layers, multi-head attention, and BERT.

liveBookmanning.com

Read Natural Language Processing in Action, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionAn attention head is an independently parameterized component of an attention layer that learns relationships among tokens from a particular perspective using its own query, key, and value matrices.

Compare how 2 books explain attention head →

Related concepts

• multi-head attentionpart of · Multi-head attention is an attention mechanism that computes multiple attention operations in parallel using distinct learned projections and combines their outputs.
• attentionuses · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• grouped-query attentionused by · Grouped-query attention partitions query heads into groups whose members share key and value projections, reducing attention computation and memory use.
• queryproduces · An attention query is a vector representation of the information sought by an input position, compared with key representations to determine which information to attend to.
• matrix-matrix multiplicationuses · Matrix multiplication combines compatible matrices to form a result whose entries are sums of products between rows of the first matrix and columns of the second.

This explainer was generated by Manning's knowledge graph from the text of Natural Language Processing in Action, Second Edition. The section links point to the parts of the book it draws on.