Multi-head attention

As explained in Inside Deep Learning: Math, Algorithms, Models

An attention mechanism that uses multiple attention heads to attend to different contexts or aspects of the input in parallel.

Multi-head attention (MHA) extends attention by learning several contexts in parallel. Each head can focus on a different pattern, then the model mixes their answers into one output. section 12.5.1

Represent the mechanism as queries over paired keys and values:

• Use key and value tensors K and V with shape (T, D), where each key has a corresponding value. section 12.5.1
• Use a query tensor Q with its own length T′; each query receives one D-dimensional response. section 12.5.1
• Apply softmax and learned nn.Linear layers so each head selectively focuses on parts of the input. section 12.5.1
• Concatenate the head outputs and project them through a final linear layer; increasing the head count does not increase output width. section 12.5.1

A PyTorch implementation pattern

Create one query, key, and value projection per head, compute Attention(wq(Q), wk(K), wv(V)) for every head, concatenate the results, and apply wo to return D features. A useful intuition is the contrast with a Python dictionary: MHA lets every key contribute according to similarity instead of requiring an exact match. section 12.5.1

Figure 12.5 Outline of the multiheaded attention block. There are three sequences as inputs: the queries Q and the keys and values K, V, which are paired together. The MHA returns a D-dimensional vector that answers each query q_i ∈ Q.

Shows Q, K, and V as three input sequences and the D-dimensional response returned for each query. section 12.5.1

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes

Explains tensor roles, shapes, head behavior, and PyTorch construction, while deferring detailed transformer additions.

Examples

Uses negation detection, a key-value dictionary analogy, tensor shapes, and nn.ModuleList projection code.

What it assumes

Assumes familiarity with attention, softmax, nn.Linear, and tensor dimensions while clarifying queries, keys, and values.

What you can do after

Apply it via PyTorch by projecting Q, K, and V per head, combining head outputs, and returning fixed-width representations.

How it connects

Connects attention mechanisms, transformers, positional encoding, recurrent neural networks, and convolutional layers.

liveBookmanning.com

Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionMulti-head attention computes multiple attention representations in parallel and combines them, potentially using distinct query and source sequences.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)An attention mechanism that computes attention through multiple separate query, key, and value subspaces to capture different relationships simultaneously.
• 

  liveBookmanning.com
  Transformers in ActionMultihead attention is an attention mechanism that computes multiple attention patterns in parallel so a model can represent different relationships among sequence positions.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA self-attention variant that splits representations into multiple attention heads to capture different relationships in parallel.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionAn attention variant that runs several attention computations in parallel with different parameters to learn different relationships.
• 

  liveBookmanning.com
  Graph Neural Networks in ActionAn attention technique that computes multiple attention operations in parallel and combines their outputs.
• 

  liveBookmanning.com
  Build a Large Language Model (From Scratch)An attention technique that computes several independent attention heads in parallel and combines their outputs to capture different relationships in the same sequence.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionMultiHeadAttention is a Keras layer that computes attention through multiple parallel attention heads and combines their outputs.
• 

  liveBookmanning.com
  Time Series Forecasting Using Foundation ModelsAn attention mechanism with multiple parallel attention heads that combines information from different representation subspaces.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthMultihead attention runs multiple attention mechanisms in parallel and combines their output representations.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionAn attention mechanism that runs several attention heads in parallel, each learning different projections and relationships.
• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsMulti-head self-attention computes several attention operations in parallel so a sequence representation can capture different relationships among its elements.
• 

  liveBookmanning.com
  The AI Pocket BookA transformer attention mechanism that splits representations into multiple heads to attend to different relationships in parallel.

Compare how 14 books explain multi-head attention →

Related concepts

• graph attention networkused by · A graph attention network is a graph neural network that uses attention weights to aggregate information from neighboring nodes, allowing their contributions to differ.
• grouped-query attentionequivalent to · Grouped-query attention partitions query heads into groups whose members share key and value projections, reducing attention computation and memory use.
• attentionkind of · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• attention headhas part · An attention head is an independently parameterized attention computation that evaluates relationships among input elements and combines information from them into a contextualized representation.
• transformer architectureused by · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• encoderpart of · An encoder is a component that transforms input data into an encoded representation for subsequent processing, reconstruction, comparison, or transmission.

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.