liveBook · Manning

Truncation

As explained in Designing AI Systems

Truncation is a context-management technique that removes older or excess messages so the remaining content fits within a token budget.

Truncation manages a limited context window by retaining the newest messages that fit a token budget and discarding older content. It is fast, predictable, and requires neither additional LLM calls nor external services, making it suitable for transactional interactions such as password resets or self-contained FAQ questions. Its production risk is selective amnesia: the discarded messages may contain facts that remain important to the current response. section 4.7.2 section 4.8

Implement truncation as a deliberate baseline, then test whether conversation behavior requires stronger context management:

  • Work backward from the latest message, accumulate tokens until the budget is reached, and drop everything older. section 4.7.2
  • Prefer it when questions are largely independent and older conversation details do not affect safety or correctness. section 4.7.6
  • Replace or supplement it with summarization when conversations exceed the budget but older flow still matters. section 4.7.6
  • Add model-managed memory for critical facts and retrieval-augmented memory when users refer to previous sessions. section 4.7.6

When the boundary removes the wrong fact

A healthcare conversation may place a penicillin allergy in message 1, symptoms in message 5, and a preferred pharmacy in message 12. If truncation keeps only messages 31–50, all three facts disappear, allowing a later response to contradict earlier information or miss safety-critical context. section 4.7.2

Figure 4.7 Simple truncation drops older conversations from the context window. Messages 1-30 are dropped and only messages 31-50 are kept. Message 1 contains "I have a penicillin allergy", which is now lost, and model loses that information when answering the user’s question.

Use the figure to inspect how a fixed recent-message window removes earlier conversation facts. section 4.7.2

How Designing AI Systems covers it

How deep it goes
Explains backward token accumulation and its information-loss risks, but defers richer memory implementation to other strategies.
Examples
Uses password resets, FAQ lookups, shopping conversations, and a healthcare allergy scenario to contrast independent and context-dependent interactions.
What it assumes
Assumes awareness of token budgets and conversation history while treating context loss and alternative strategies as design decisions.
What you can do after
Apply it via the Session Service SDK by selecting a truncation strategy and token or message limit.
How it connects
Connects context windows with summarization, model-managed memory, retrieval-augmented memory, the Session Service, and the Data Service.

Also explained in

Compare how 3 books explain truncation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.