liveBook · Manning

Failure mode

As explained in Retrieval Augmented Generation, The Foundational Ideas

A failure mode is a specific way a system or component can behave incorrectly or fail to meet its requirements.

A financial-services assistant may answer a mortgage-rate question correctly yet fail on a policy question. The policy might be absent, buried beneath outdated documents, removed from the supplied context, or ignored in the answer. Each is a different failure mode: a specific way the system fails to meet its goal. Naming the mode turns a vague “accuracy problem” into something that can be tested and fixed. section 1.4

RAG failures can occur during retrieval, context augmentation, or generation. Seven documented failure points distinguish missing content, poor ranking, factual contradiction, relevant information excluded from context, information present but unused, incorrect specificity, and incomplete answers. The distinction matters: improving ranking will not make a generator use evidence it already has. section 1.4 section 15.5.1

Table 1.1 RAG failure points
Failure point
Name
 Description
Impact on financial services example
 FP1  
Missing content 
Answer doesn’t exist in the knowledge base 
Policy questions about procedures not yet documented 
 FP2  
Missed the top rank 
Correct document retrieved, but ranked too low 
Current guidelines are buried beneath outdated policies 
 FP3  
Factually inconsistent hallucination 
Model contradicts the retrieved information 
Fabricated compliance procedures despite the correct source documents 
 FP4  
Not in context 
Relevant chunk retrieved but filtered out 
Important policy details were removed during context optimization 
 FP5  
Not extracted 
Information in context, but not used in response 
Model ignores the retrieved compliance requirements 
 FP6  
Incorrect specificity 
Wrong level of detail for the user’s needs 
Strategic overview of when operational steps are needed 
 FP7  
Incomplete 
Partial answer when a complete response is needed 
Missing critical steps in multi-part procedures 

Compare the named failure points with their effects on financial-services questions before choosing a fix. section 1.4

Diagnose before adding components

If the current guideline is retrieved but ranked beneath old policies, investigate ranking (FP2); RAG-Fusion is one possible remedy. If correct compliance documents are retrieved but the answer invents a procedure (FP3), a retrieval audit such as CRAG targets a different problem. First establish an evaluation baseline, then add a component only when a measured failure warrants its latency and cost. section 1.4 section 1.0 section 1.1 section 15.5.1

Some failures require more than a better single retrieval pass. Multi-hop questions need facts chained across steps; temporal questions need facts aligned in time. Errors also occur across chunking, retrieval, reranking, and generation, so an error breakdown from another system is not a substitute for measuring your own. section 4.5.5

Table 15.1 RAG techniques by complexity and performance impact
Technique
Complexity
Primary failure points addressed
When to use
Naive RAG (ch. 4)
Low
Baseline
Simple factual queries over clean corpora
Hybrid search (ch. 2, 8)
Low
FP1, FP2
Queries mixing keywords and concepts
Reranking (ch. 8)
Low-Medium
FP2, FP4
Retrieval noise degrades generation
HyDE (ch. 7)
Medium
FP1, FP2
Abstract or underspecified queries
RAG-Fusion (ch. 8)
Medium
FP1, FP2, FP7
Diverse query interpretations needed
RAPTOR (ch. 10)
Medium
FP5, FP6
Large documents; queries at varying abstraction levels
Context compression (ch. 13)
Medium
FP4, FP5
Long or noisy retrieved contexts
Self-RAG (ch. 9)
High
FP3, FP5, FP7
Hallucination-sensitive applications
FLARE (not implemented; see the ch. 11 planner)
High
FP1, FP7
Long-form answers; information needs emerge mid-generation
CRAG (ch. 11)
High
FP1, FP3
Retrieval quality varies by query
LevelRAG (ch. 11)
High
FP1, FP7
Multi-part queries needing planned decomposition
Graph RAG (ch. 12)
High
FP1, FP7
Multi-hop questions over connected entities
Adaptive-RAG (ch. 15)
High
All
Mixed query complexity
Agentic RAG (ch. 15)
Very High
All
Complex multi-step queries, fragmented knowledge

Use the technique-to-failure-point comparison to weigh a proposed fix against implementation and runtime complexity. section 15.5.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines a three-layer lifecycle taxonomy and seven failure points, then links diagnosis to remedies rather than specifying a universal repair procedure.
Examples
Uses financial-services policy answers, missing or outdated guidelines, fabricated compliance procedures, and multi-hop and temporal questions.
What you can do after
Diagnose/debug with it by measuring the failing stage and selecting a proportionate remedy instead of adding complexity by default.
How it connects
Connects retrieval, context augmentation, generation, reranking, hallucination, evaluation, and iterative retrieval.

Also explained in

Compare how 6 books explain failure mode →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.