liveBook · Manning

Hierarchical memory

As explained in Designing AI Systems

Hierarchical memory is a memory architecture that organizes information into layers with different levels of structure, compression, recency, and retention priority.

Hierarchical memory allocates different retention and compression policies to different kinds of conversation context. Critical facts remain structured, older history becomes a rolling summary, and recent exchanges stay verbatim, preserving fidelity where it matters while controlling context-window cost. section 4.7.4 section 4.8

A three-tier design separates durable facts from narrative context and current conversational detail:

  • Model-managed memories store critical facts as structured key-value records, such as allergies, preferences, or named providers. section 4.7.4
  • A rolling summary compresses older conversation into a compact narrative covering the situation, attempted actions, and concerns. section 4.7.4
  • Recent messages remain verbatim so follow-up questions retain the nuance and flow of the active topic. section 4.7.4

The hierarchy addresses limitations of simple truncation, which preserves recency but can discard important early information. It can also coexist with retrieval-augmented memory, where prior conversations become searchable through a Data Service. Context assembly should account for attention patterns that favor information near the beginning and end of the input. section 4.8 section 4.7.4

Figure 4.9 Hierarchical memory combines three tiers of context. Model-managed memories store critical facts as structured key-value pairs. A rolling summary compresses older conversation history into a compact narrative. Recent messages remain verbatim to preserve nuance in the current exchange.

Use the three-tier diagram to compare structured facts, rolling summaries, and verbatim recent messages. section 4.7.4

How Designing AI Systems covers it

How deep it goes
Explains three retention tiers and assembly considerations, but does not specify a concrete storage schema or implementation.
Examples
Uses patient facts, rolling clinical conversation summaries, recent exchanges, and prior-session search as illustrative artifacts.
What it assumes
Assumes awareness of context-window limits and teaches how truncation, summarization, structured memory, and retrieval divide responsibility.
What you can do after
Operate/configure it by selecting truncation, hierarchical summarization, or retrieval-augmented strategies for different conversation risks.
How it connects
Connects model-managed memories, summarization, truncation, retrieval-augmented memory, context engineering, and LLM attention patterns.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.