liveBook · Manning

Self-attention

As explained in Retrieval Augmented Generation, The Foundational Ideas

Self-attention is an attention mechanism that computes relationships among tokens by allowing each token to incorporate information from other tokens in the same sequence.

A fixed word vector cannot distinguish a river bank from a savings bank. Self-attention gives a Transformer a way to relate a word to others in the same sequence: nearby clues such as “river” or “money” help produce a context-dependent representation. BERT uses this capability to represent the meaning of a sentence rather than treating each word as having one fixed meaning. section 2.3

Matching support tickets

A support platform may need to match “My internet connection keeps dropping” with resolved tickets about network stability. Contextual representations help capture that similarity, but using BERT to score every new–resolved ticket pair requires a separate model pass for each pair. Contextual understanding and efficient search are different problems. section 2.3

Self-attention also shapes retrieval architecture. Its compute and memory costs grow quadratically with sequence length, O(n²). Concatenating dozens of retrieved documents into one prompt can therefore become expensive; long prompts can also bury useful evidence in the middle. Fusion-in-Decoder (FiD) instead encodes each document–query pair separately, then lets the decoder attend across their combined representations. For similarly sized documents, this makes the encoding cost grow roughly linearly with the number of documents, at the price of a more involved architecture. section 5.1 section 6.2

Table 5.1 Classic FiD vs. big context
Feature
Classic FiD (Encoder-Decoder)
Long-Context RAG (Decoder-Only)
Architecture
T5 / ATLAS. Encodes each document independently, then fuses them only during generation.
GPT / Llama. Concatenates all documents into one giant sequence and processes them simultaneously.
Complexity
Linear. Encoding 100 documents costs roughly 10 times the compute of encoding 10.
Quadratic. Self-attention costs grow squarely with length. Encoding 100 documents is significantly more expensive than encoding 10.
Lost in the Middle
Low Risk. Because the encoder processes every document in isolation, it creates a distinct representation for every passage before fusion occurs.
High Risk. Large Decoder-only models often ignore information buried in the middle of a massive prompt context.
Implementation
Harder. Requires fine-tuning and hosting your own model (e.g., HuggingFace transformers).
Easier. Often just an API call or a prompt template change.
Best For
Scale and Efficiency. When you have a fixed domain (e.g., medical records) and need to synthesize answers from 50–100+ documents cheaply and accurately.
Flexibility and Speed. When you need to build a system today that handles general queries over a few dozen documents without training.

Compare independent document encoding with one concatenated long context, including compute, implementation, and the risk of missing middle passages. section 5.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains contextual word relationships and quadratic sequence-length cost, but does not derive the attention calculation.
Examples
Illustrates context with two meanings of “bank” and retrieval tradeoffs with support tickets and multiple retrieved documents.
What it assumes
Introduces the mechanism through fixed word vectors and contextual meaning, without requiring a formal attention derivation.
What you can do after
Evaluate/understand when contextual representations help and when long-input self-attention costs favor independent document encoding.
How it connects
Connects self-attention to Transformers, BERT, Sentence-BERT, long-context RAG, Fusion-in-Decoder, and cross-attention.

Also explained in

Compare how 5 books explain self-attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.