liveBook · Manning

Lost in the middle

As explained in Designing AI Systems

The lost in the middle phenomenon is a language-model behavior in which information positioned in the middle of a long context receives less attention than information near the beginning or end.

Lost in the middle describes a language-model behavior: performance is strongest when relevant information appears near the beginning or end of a long context, and degrades when key information sits in the middle. section 4.7.4

Treat the phenomenon as a context-assembly constraint rather than a defect to eliminate. Place critical facts, memories, and safety information first; place recent messages last; reserve the middle for content that can tolerate weaker attention. section 4.7.4 section 10.8

Allocate the context budget according to the application’s information priorities:

  • Start with roughly 10% for memories, 20% for a rolling summary, and 70% for recent messages. section 4.7.4
  • Increase the memory allocation when many critical facts must remain visible, such as in healthcare assistance. section 4.7.4
  • Increase summary space for long research conversations or recent-message space for coding discussions focused on the latest exchanges. section 4.7.4
  • Adjust documents and history according to the application: grounding-heavy assistants may prioritize documents, while therapy assistants may prioritize conversation history. section 10.8

Three-tier conversation context

A hierarchical memory design places structured critical facts first, compresses older exchanges into a rolling summary, and keeps the latest ten or twenty exchanges verbatim at the end. The middle therefore contains compressed history rather than information essential to the immediate response. section 4.7.4

Figure 4.9 Hierarchical memory combines three tiers of context. Model-managed memories store critical facts as structured key-value pairs. A rolling summary compresses older conversation history into a compact narrative. Recent messages remain verbatim to preserve nuance in the current exchange.

Use the figure to inspect how model-managed memories, a rolling summary, and verbatim recent messages occupy different context tiers. section 4.7.4

Figure 10.8 Context window assembly. Six sources compete for space within the model's context window. The assembly strategy determines what Claw knows when generating a response.

Use the context-window diagram to trace how multiple sources compete for space and how assembly order determines what the model receives. section 10.8

How Designing AI Systems covers it

How deep it goes
Explains the observed attention pattern and translates it into ordering and budget decisions, without detailing model internals.
Examples
Uses structured memories, rolling summaries, recent verbatim messages, and document-versus-history allocation examples.
What you can do after
Apply it via context assembly by ordering sources and tuning their token-budget shares for the application.
How it connects
Connects hierarchical memory, context engineering, token budgets, summaries, and conversation history.

Also explained in

Compare how 3 books explain lost in the middle →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.