A fixed word vector cannot distinguish a river bank from a savings bank. Self-attention gives a Transformer a way to relate a word to others in the same sequence: nearby clues such as “river” or “money” help produce a context-dependent representation. BERT uses this capability to represent the meaning of a sentence rather than treating each word as having one fixed meaning. section 2.3
Matching support tickets
A support platform may need to match “My internet connection keeps dropping” with resolved tickets about network stability. Contextual representations help capture that similarity, but using BERT to score every new–resolved ticket pair requires a separate model pass for each pair. Contextual understanding and efficient search are different problems. section 2.3
Self-attention also shapes retrieval architecture. Its compute and memory costs grow quadratically with sequence length, O(n²). Concatenating dozens of retrieved documents into one prompt can therefore become expensive; long prompts can also bury useful evidence in the middle. Fusion-in-Decoder (FiD) instead encodes each document–query pair separately, then lets the decoder attend across their combined representations. For similarly sized documents, this makes the encoding cost grow roughly linearly with the number of documents, at the price of a more involved architecture. section 5.1 section 6.2
Table 5.1 Classic FiD vs. big context
|
Feature
|
Classic FiD (Encoder-Decoder)
|
Long-Context RAG (Decoder-Only)
|
|
Architecture
|
T5 / ATLAS. Encodes each document
independently, then fuses them only during generation.
|
GPT / Llama. Concatenates all
documents into one giant sequence and processes them simultaneously.
|
|
Complexity
|
Linear. Encoding 100 documents
costs roughly 10 times the compute of encoding 10.
|
Quadratic. Self-attention costs
grow squarely with length. Encoding 100 documents is significantly more
expensive than encoding 10.
|
|
Lost in the Middle
|
Low Risk. Because the encoder
processes every document in isolation, it creates a distinct representation
for every passage before fusion occurs.
|
High Risk. Large Decoder-only
models often ignore information buried in the middle of a massive prompt
context.
|
|
Implementation
|
Harder. Requires fine-tuning and
hosting your own model (e.g., HuggingFace transformers).
|
Easier. Often just an API call or
a prompt template change.
|
|
Best For
|
Scale and Efficiency. When you
have a fixed domain (e.g., medical records) and need to synthesize answers
from 50–100+ documents cheaply and accurately.
|
Flexibility and Speed. When you
need to build a system today that handles general queries over a few dozen
documents without training.
|
Compare independent document encoding with one concatenated long context, including compute, implementation, and the risk of missing middle passages. section 5.1




