liveBook · Manning

Summarization

As explained in Designing AI Systems

Summarization is the generation of a shorter representation that preserves the salient information from a longer text or interaction.

Summarization compresses older conversation messages into a shorter representation so essential facts remain available when recent messages are kept verbatim. It trades an LLM call’s latency and cost for a longer effective context than simple truncation provides. section 4.7.3

Preserve the information the application needs, and make summary generation operationally efficient:

  • Use a task-specific prompt: healthcare summaries should retain conditions, allergies, and safety concerns; other domains may need commitments, deadlines, or agreed terms. section 4.7.3
  • Mark compressed content clearly as a summary, cache it, and regenerate it only when the messages it covers change; a message hash can track that set. section 4.7.3
  • Use a fast, inexpensive model to control summarization costs. section 4.7.3

Conversation history

Compress a long healthcare conversation into a brief summary retaining a patient’s penicillin allergy, knee pain, and preference for morning appointments. Pair that summary with recent verbatim messages; the older back-and-forth is no longer available word for word. section 4.7.3

Figure 4.8 Summarization compresses older messages while preserving key information. Instead of discarding messages 1-30 entirely, an LLM call compresses them into a summary that captures critical facts: the penicillin allergy, insurance verification, appointment preferences. Messages 31-50 remain verbatim. The model now sees the summary followed by recent messages, retaining the context that simple truncation would have lost.

Shows older messages compressed into a summary while recent messages remain verbatim, preserving key facts that truncation would discard. section 4.7.3

How Designing AI Systems covers it

How deep it goes
Explains prompt design, summary labeling, caching, and model choice, but does not specify when to trigger summarization.
Examples
Uses a healthcare conversation retaining an allergy, symptom, and appointment preference as its example.
What you can do after
Apply summarization to retain selected facts while reducing conversation-history tokens.
How it connects
Connects conversation history compression with truncation, caching, and context management.

Also explained in

Compare how 17 books explain summarization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.