Evaluation turns RAG reliability into an engineering practice: measure behavior against defined criteria, connect scores to failure points, and decide whether added complexity is worth its cost. section 1.1 section 1.4
Use reference-free measures to inspect different parts of the pipeline:
- Faithfulness checks whether answer claims are supported by retrieved passages. section 1.1
- Answer relevance checks whether the response addresses the user’s question. section 1.1
- Context relevance checks how much retrieved material is useful. section 1.1
- Targeted test suites expose failure points such as missing content, ranking errors, hallucinations, filtered context, and unused evidence. section 1.4
Putting a price on complexity
An agentic support assistant gained 8.23 points of retrieval accuracy but incurred 6.3 times the latency. Evaluation therefore supports a selection decision rather than assuming that a more advanced architecture is automatically better. section 1.1 section 15.1
Table 1.1 RAG failure points
|
Failure point
|
Name
|
Description
|
Impact on financial services example
|
|
FP1
|
Missing content
|
Answer doesn’t exist in the knowledge base
|
Policy questions about procedures not yet documented
|
|
FP2
|
Missed the top rank
|
Correct document retrieved, but ranked too low
|
Current guidelines are buried beneath outdated
policies
|
|
FP3
|
Factually inconsistent hallucination
|
Model contradicts the retrieved information
|
Fabricated compliance procedures despite the correct
source documents
|
|
FP4
|
Not in context
|
Relevant chunk retrieved but filtered out
|
Important policy details were removed during context
optimization
|
|
FP5
|
Not extracted
|
Information in context, but not used in response
|
Model ignores the retrieved compliance requirements
|
|
FP6
|
Incorrect specificity
|
Wrong level of detail for the user’s needs
|
Strategic overview of when operational steps are
needed
|
|
FP7
|
Incomplete
|
Partial answer when a complete response is needed
|
Missing critical steps in multi-part procedures
|
Map a poor score to the relevant RAG failure point before choosing a corrective technique or architectural change. section 1.4




























