Rolling conversation context keeps only a bounded portion of prior messages in each model request. Retaining recent messages preserves relevant context while reducing the risk of exceeding provider token limits. section 8.2.2
Implement the technique by:
- Pairing a message-history limit with maxTokens so the request leaves room for the generated response. section 8.2.2
- Removing the oldest messages when the history exceeds the chosen bound, such as the last 10 messages. section 8.2.2
- Monitoring token counts with js-tiktoken when supported, while recognizing that some models may require a custom solution. section 8.2.2
