A context window limits how much input a model can process at once. In a retrieval-augmented system, fitting documents into that window is not enough: key facts buried among long, loosely related passages may be used less reliably, increasing the risk of an unsupported answer. section 3.6.5
Figure 3.7 The limits of the context window across LLMs
The illustrated “Lost in the Middle” pattern shows why placement matters: information near the beginning or end of a long context is used more reliably than information in the middle. section 3.6.5
Keep the answer-bearing passage
For “How do I set up international roaming?”, a telecom retriever may find an activation FAQ, a price list, a travel-coverage post, and service terms. Only the FAQ provides the requested steps. Filtering by semantic relevance keeps the input focused instead of concatenating every document that mentions roaming. section 3.6.5
Treat input length as a pipeline constraint, not a target to fill.
- Set a minimum semantic-similarity threshold before passing retrieved passages to the answering model; exclude material that is related but unnecessary. section 3.6.5
- Check the embedding model’s separate maximum input length when indexing long documents. Truncation can discard vital information, so consider chunking. section 4.2.4
- When one assistant must handle unrelated tasks and its context fills with irrelevant information, specialized agents with an orchestrator are another architectural option. section 8.0
