liveBook · Manning

Context window

As explained in Designing AI Systems

A context window is the bounded amount of input and prior interaction content that a language model can process for one generation.

A context window is the bounded amount of input and prior interaction content a language model can process for one generation. System prompts, conversation history, tool definitions, and retrieved documents compete for this capacity, so storing complete history does not guarantee that every message can be sent to the model. section 4.0 section 4.7

Production systems manage context windows through complementary controls:

  • Discover model capabilities at runtime and filter for a required context-window size instead of hardcoding model names. section 3.1.2
  • Persist complete conversation history, then select only the messages and memories needed for the next model call. section 4.0 section 4.7
  • Use a sliding window or other strategy when history exceeds available capacity, accepting that recent-message retention can omit early preferences or facts. section 10.3

Selecting a model and limiting session history

A workflow can choose a model whose context_window exceeds 100,000 tokens for a 50-page document. For an extended conversation, it can retrieve only the most recent 50 messages while retaining the full history in persistent storage. section 3.1.2 section 10.3

Listing 3.2 Model discovery response with capabilities
Response:
  models: [
    {
      name: "gpt-4o",
      provider: "openai",
      context_window: 128000,    #A
      supports_vision: true,
      supports_tools: true
    },
    {
      name: "claude-sonnet-4-5",
      provider: "anthropic",
      context_window: 200000,
      supports_vision: true,
      supports_tools: true
    },
    {
      name: "llama-3-70b-medical",    #B
      provider: "internal",
      context_window: 8192,
      supports_vision: false,
      supports_tools: false
    }
  ]
 

Inspect the model-discovery response to see context_window represented alongside provider and other capabilities. section 3.1.2

Listing 10.3 Managing context window limits
history, _ = platform.sessions.get_messages(    #A
    session_id=session.session_id, limit=50    #A
)    #A

Inspect the minimal history limit used to prevent context-window overflow while preserving the complete conversation in storage. section 10.3

How Designing AI Systems covers it

How deep it goes
Explains capacity constraints through model metadata and a sliding-window example, while deferring broader strategy implementation.
Examples
Uses a 50-page document, patient intake with a penicillin allergy, and a 50-message history limit.
What you can do after
Operate model selection and session-history assembly using capability filters and bounded message retrieval.
How it connects
Connects model discovery, session services, conversation history, memory, prompt engineering, and retrieval-augmented approaches.

Also explained in

Compare how 38 books explain context window →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.