liveBook · Manning

Context window

As explained in Retrieval Augmented Generation, The Foundational Ideas

Context length is the maximum amount of input text that a model can process in one input sequence.

A context window is the maximum amount of input text a model can process in one input sequence. Larger windows let models reason across more retrieved documents, but they still cannot accept unlimited material. section 1.3.4 section 4.7.2

Longer context does not eliminate retrieval. Feeding an entire document collection into a large prompt can obscure relevant facts and produce “lost in the middle” effects, especially when the useful information is only a small fraction of the available content. section 1.3.4

Context size creates practical trade-offs:

  • Extended windows support reasoning across multiple retrieved documents. section 1.3.4
  • Capacity remains finite even at 32,000, 100,000, or approximately one million tokens. section 4.7.2
  • Retrieval-first systems preserve efficiency and precision when relevant information is sparse within a large knowledge base. section 1.3.4
  • Chunk-based retrieval can underperform long context, while summarization-based retrieval can match it and retrieval retains advantages for dialogue and general queries. section 4.7.2

Choosing between a large window and retrieval

For a large knowledge base, placing the entire collection into a million-token prompt may satisfy capacity requirements but still bury the answer. A retrieval pipeline instead selects relevant material before generation; summarization-based retrieval can preserve long-context performance, while chunk boundaries may weaken results. section 1.3.4 section 4.7.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines finite capacity and explains retrieval trade-offs, but stops short of implementation details for managing context.
Examples
Uses 32,000-, 100,000-, and million-token windows, large knowledge bases, retrieved documents, and lost-in-the-middle behavior.
What you can do after
Evaluate whether longer context or retrieval better fits a knowledge-intensive, dialogue, or general-query workload.
How it connects
Connects context windows to retrieval-augmented generation, context engineering, long-context models, summarization, and chunking.

Also explained in

Compare how 38 books explain context window →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.