liveBook · Manning

Semantic flattening

As explained in Retrieval Augmented Generation, The Foundational Ideas

Semantic flattening is the loss of document-level structure when a corpus is represented as independent, disjoint vector items.

Semantic flattening occurs when a document’s hierarchy and relationships are lost as it is represented by independent chunks or vectors. Retrieval may find relevant fragments while missing the context that connects them into a broader answer. section 10.0 section 10.1.1

When the relationship is the answer

Suppose Q2 risks appear in one document and Q3 risks in another. Retrieving both snippets may not reveal how risks changed: the trend is a relationship between chunks, not a fact stated in either one. The system can return specific instances yet miss the requested evolution. section 10.1

Flattening can cause several kinds of retrieval failure:

  • Incorrect specificity: results match the topic but not the requested level, such as returning individual risks instead of their evolution. section 10.0
  • Incomplete answers: synthesis fails when the answer depends on narrative flow, causal links, or themes spread across fragments. section 10.0
  • Lost context: isolated terms can be ambiguous, and chunk boundaries can separate an argument or a definition from its application. section 10.1

Better chunk boundaries can help, but fixed-size or sentence-based chunking still returns a flat list. RAPTOR instead recursively clusters chunks and summarizes them, creating representations at multiple levels of abstraction. section 10.1.1 section 10.0

Figure 10.1 The semantic flattening problem.

Compare the document’s connected hierarchy with the flat vector index to see how relationships between fragments disappear. section 10.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains chunking, context loss, and retrieval failures, then contrasts flat chunks with RAPTOR’s hierarchical summaries.
Examples
Uses quarterly risk trends, ambiguous meanings of “bank,” sarcasm, and a historian’s archive analogy.
What it assumes
Assumes familiarity with chunked documents and vector retrieval, while explaining the structural problem through concrete analogies.
What you can do after
Evaluate retrieval failures where answers depend on context or relationships across chunks.
How it connects
Connects semantic flattening to RAG, chunking, vector stores, and RAPTOR.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.