An evaluation suite is a collection of tests that checks RAG behavior against known failure conditions and catches regressions before deployment. Targeting specific failure points turns diagnosis into a repeatable engineering practice. section 1.4
Table 1.1 RAG failure points
|
Failure point
|
Name
|
Description
|
Impact on financial services example
|
|
FP1
|
Missing content
|
Answer doesn’t exist in the knowledge base
|
Policy questions about procedures not yet documented
|
|
FP2
|
Missed the top rank
|
Correct document retrieved, but ranked too low
|
Current guidelines are buried beneath outdated
policies
|
|
FP3
|
Factually inconsistent hallucination
|
Model contradicts the retrieved information
|
Fabricated compliance procedures despite the correct
source documents
|
|
FP4
|
Not in context
|
Relevant chunk retrieved but filtered out
|
Important policy details were removed during context
optimization
|
|
FP5
|
Not extracted
|
Information in context, but not used in response
|
Model ignores the retrieved compliance requirements
|
|
FP6
|
Incorrect specificity
|
Wrong level of detail for the user’s needs
|
Strategic overview of when operational steps are
needed
|
|
FP7
|
Incomplete
|
Partial answer when a complete response is needed
|
Missing critical steps in multi-part procedures
|
Use the failure-point taxonomy to choose what the suite should test, from missing content and ranking errors to unsupported answers. section 1.4
A production RAG suite can test the system at three levels:
- Component tests isolate stages such as retrieval, reranking, and generation; run them on every code change. section 15.6.4
- Integration tests exercise the full pipeline on an evaluation dataset before deployment, exposing interactions between stages. section 15.6.4
- Regression tests compare the current pipeline with the previous production version on challenging or previously failed queries. Block deployment when a known-bad case worsens. section 15.6.4
Keep the test set current by adding production failures, preserving queries the system handled well, and including synthetic and expert-authored questions. Include unanswerable questions to test whether the system recognizes the limits of its knowledge base. section 14.4.2 section 15.6.4
Checking a pipeline change
After changing chunk size, run integration tests to catch effects across retrieval and generation. Then compare a curated set of reference answers to check that queries the system previously answered correctly have not regressed. section 15.6.4 section 14.2.6





















