Truncation removes tokens that exceed a model’s context window. Because the window includes both prompt and response tokens, earlier conversation or input can disappear without warning, leaving the model inconsistent with prior exchanges. section 1.1 section 1.3.2
The overheard conversation
Imagine someone hearing only the last two sentences of a long conversation and then offering advice. When truncation removes earlier tokens, an AI can respond similarly: it may lose crucial instructions, context, or large portions of supplied code. section 1.3.2
Reduce the risk of losing important context by:
- Estimating token usage and remembering that tokenization differs between models. section 1.1 section 1.3.2
- Chunking long inputs into segments that fit the context window while retaining enough local context. section 1.3.2
- Using sliding windows, iterative refinement, summaries, or external memory to preserve relevant information across segments. section 1.1 section 1.3.2
