liveBook · Manning

Context window

As explained in Build an Advanced RAG Application (From Scratch)

Context length is the maximum amount of input context, typically measured in tokens, that a model can process for an operation.

A context window is the maximum token count an embedding model can process in one forward pass. Inputs beyond that limit may be ignored, so long documents must be handled within the model’s supported window. section 6.3 section 6.3.1

For embedding models, context-window limits create three practical constraints:

  • Nomic Embed Text supports up to 8,192 tokens, roughly 6,000–7,000 English words depending on vocabulary and tokenization. section 6.3.1
  • Transformer self-attention compares every token with every other token, giving processing costs that grow quadratically with sequence length. section 6.3.1
  • Increasing the limit is not merely expensive: embeddings must compress more diverse information into the same fixed-size vector. section 6.3.2

Scaling the window

At 8,192 tokens, attention requires roughly 67 million computations. Doubling the window to 16,384 tokens raises that figure to approximately 268 million; 32,768 tokens exceeds one billion. At dataset scale, the resulting GPU requirements and processing costs make routine long-document embedding impractical. section 6.3.1 section 6.3.2

Figure 6.3 Computational Cost of Context Window Expansion

Use the computational-cost figure to compare how attention operations escalate as the context window expands. section 6.3.2

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Explains token limits, self-attention scaling, and fixed-vector compression, but stops short of specifying a chunking algorithm.
Examples
Quantifies attention costs at 8,192, 16,384, and 32,768 tokens and relates them to large document datasets.
What you can do after
Apply it via document chunking when preparing long documents for embedding and retrieval.
How it connects
Connects context windows to embedding models, self-attention, document chunking, vector databases, and retrieval-augmented generation.

Also explained in

Compare how 38 books explain context window →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.