liveBook · Manning

Context window

As explained in AI Model Evaluation

A context window is the bounded amount of input and generated text that a language model can process in one invocation.

A context window is the token capacity available to an LLM for the input and generated text of a request. In practice, context size includes the prompt, conversation history, retrieved material, and other input sent to the model. As conversations grow or prompts become more elaborate, context size can increase latency and cost while leaving less room for useful output. section 4.2.3 section 9.10.1

Treat context size as an engineering constraint:

  • Measure tokens sent to the model alongside prompt-processing time, time to first token, total latency, and cost per request. section 4.2.3 section 4.6
  • Trim or restructure overly long prompts when they add latency, increase evaluation cost, or crowd out important context. section 9.10.1
  • Choose an evaluator with a context window large enough to see the evidence required for a long-form judgment. section 9.5.4
  • Track context size separately from retrieval latency, because retrieved material can expand the request while retrieval creates its own bottleneck. section 4.2.3

Prompt debugging

A judge prompt that includes extensive instructions, source material, and evaluation history may become expensive and slow. Inspect its token usage, cap generated output where supported, and test missing, conflicting, or noisy context to confirm predictable failure behavior. section 9.10.1

How AI Model Evaluation covers it

How deep it goes
Explains operational effects and measurement, but does not detail model-specific context limits or tokenizer internals.
Examples
Uses growing conversations, long-form evaluation evidence, oversized prompts, and noisy or conflicting context as examples.
What it assumes
Assumes basic familiarity with LLM prompts, tokens, generated output, and retrieval-based systems.
What you can do after
Diagnose latency and cost pressure by measuring token usage, trimming prompts, and testing context-related failure cases.
How it connects
Ties context size to latency, cost per request, prompt processing time, retrieval latency, token efficiency, and evaluator models.

Also explained in

Compare how 38 books explain context window →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.