liveBook · Manning

Multi-head attention

As explained in Fight Fraud with Machine Learning

Multi-head attention is an attention module containing multiple self-attention layers whose outputs are combined to form a richer representation.

Multi-head attention combines multiple self-attention layers. Each layer assigns attention coefficients to input time steps, allowing different layers to extract different signals from the same sequence. section 4.3.4

Within each self-attention layer, attention weights Pq, Pk, and Pv produce coefficients c1 through cn. These coefficients control how strongly each input element contributes to each output. The layer concatenates the outputs from its multiple self-attention layers into a richer feature representation. section 4.3.4

Figure 4.21 A multi-head attention layer showing the concatenation of features produced by multiple self-attention layers, producing a set of attention coefficients that inform the model to look at different parts of the input sequence in different proportions.

Shows how multiple self-attention outputs are concatenated and attend to different input-sequence elements in different proportions. section 4.3.4

How Fight Fraud with Machine Learning covers it

How deep it goes
Defines heads, attention coefficients, and concatenation, but defers detailed transformer architecture.
Examples
Uses time-step sequences, attention weights Pq/Pk/Pv, coefficients c1–cn, and concatenated feature representations.
What it assumes
Assumes attention weights, sequential data, and recurrent layers as context for introducing self-attention-only transformers.
What you can do after
Evaluate how multiple attention heads capture different signals from a sequence without recurrent connections.
How it connects
Connects self-attention, attention mechanisms, transformers, recurrent layers, and sequential data.

Also explained in

Compare how 25 books explain multi-head attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Fight Fraud with Machine Learning. The section links point to the parts of the book it draws on.