Summarization compresses older conversation messages into a shorter representation so essential facts remain available when recent messages are kept verbatim. It trades an LLM call’s latency and cost for a longer effective context than simple truncation provides. section 4.7.3
Preserve the information the application needs, and make summary generation operationally efficient:
- Use a task-specific prompt: healthcare summaries should retain conditions, allergies, and safety concerns; other domains may need commitments, deadlines, or agreed terms. section 4.7.3
- Mark compressed content clearly as a summary, cache it, and regenerate it only when the messages it covers change; a message hash can track that set. section 4.7.3
- Use a fast, inexpensive model to control summarization costs. section 4.7.3
Conversation history
Compress a long healthcare conversation into a brief summary retaining a patient’s penicillin allergy, knee pain, and preference for morning appointments. Pair that summary with recent verbatim messages; the older back-and-forth is no longer available word for word. section 4.7.3
Figure 4.8 Summarization compresses older messages while preserving key information. Instead of discarding messages 1-30 entirely, an LLM call compresses them into a summary that captures critical facts: the penicillin allergy, insurance verification, appointment preferences. Messages 31-50 remain verbatim. The model now sees the summary followed by recent messages, retaining the context that simple truncation would have lost.

Shows older messages compressed into a summary while recent messages remain verbatim, preserving key facts that truncation would discard. section 4.7.3
















