Attention matrix

As explained in Natural Language Processing in Action, Second Edition

An attention matrix is a matrix of learned or computed weights representing how strongly elements in a sequence should attend to one another.

An attention matrix records how strongly tokens in a sequence connect to one another. Instead of processing text one token at a time, a transformer multiplies the matrix by a sequence of token embeddings, allowing connections across the sequence to contribute to its output. section 9.1.1

A short question

The tokens [What] [is] [BERT] [?] can be represented by four embedding vectors. Together they form an encoding: a two-dimensional array for the token sequence. An encoder attention matrix can connect words across that sequence, rather than looking only at neighboring tokens. section 9.1.2

Transformer layers keep their input and output encodings in a consistent shape. That lets an attention matrix span the required sequence length in each layer and lets layers stack. Input text is truncated or padded to that length, so the chosen length also limits how much text can be processed at once. section 9.1.2 section 9.1.1

Figure 9.3 The encoder attention matrix as connections between words

View the encoder attention matrix as connections between words in a sequence. section 9.1.2

Text generation needs a further constraint. During training, a decoder can receive a whole sequence at once, but a prediction must not use later tokens. A mask blocks those future connections in the attention matrix, preventing the model from peeking at the answer it is learning to generate. A causal data collator applies this earlier-token-only rule when preparing GPT-2 fine-tuning data. section 9.2.2 section 10.2.2

How Natural Language Processing in Action, Second Edition covers it

How deep it goes

Explains sequence-wide multiplication, encoding shapes, and causal masking, but does not derive how attention weights are calculated.

Examples

Uses a four-token question, an encoder word-connection diagram, and causal GPT-2 fine-tuning data.

What it assumes

Defines encodings as token-sequence embeddings and illustrates padded token sequences before discussing attention across them.

What you can do after

Evaluate/understand it by tracing which tokens an encoder can connect and why a decoder must block future tokens.

How it connects

Links attention to token embeddings, positional encoding, transformer encoders and decoders, and causal masking.

liveBookmanning.com

Read Natural Language Processing in Action, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)An attention map is a matrix of attention probabilities showing how strongly tokens attend to one another.
• 

  liveBookmanning.com
  Machine Learning System DesignAn attention matrix is a two-dimensional array of attention weights relating positions in one sequence to positions in another sequence.

Compare how 3 books explain attention matrix →

Related concepts

• inner productproduced by · An inner product is a scalar-valued operation on two vectors that combines their corresponding components, summing their products and, for complex vectors, using conjugation.
• Vision Transformerproduced by · A Vision Transformer is a Transformer-based neural-network architecture that processes images as sequences of tokenized regions using attention.
• attention weighthas part · An attention weight is a computed coefficient, often normalized, that determines how strongly an input element or value contributes to an attention-based representation.
• attention headproduced by · An attention head is an independently parameterized attention computation that evaluates relationships among input elements and combines information from them into a contextualized representation.

This explainer was generated by Manning's knowledge graph from the text of Natural Language Processing in Action, Second Edition. The section links point to the parts of the book it draws on.