liveBook · Manning

Test suite

As explained in Retrieval Augmented Generation, The Foundational Ideas

A test suite is a collection of tests designed to assess specified behaviors or failure conditions in a system.

An evaluation suite is a collection of tests that checks RAG behavior against known failure conditions and catches regressions before deployment. Targeting specific failure points turns diagnosis into a repeatable engineering practice. section 1.4

Table 1.1 RAG failure points
Failure point
Name
 Description
Impact on financial services example
 FP1  
Missing content 
Answer doesn’t exist in the knowledge base 
Policy questions about procedures not yet documented 
 FP2  
Missed the top rank 
Correct document retrieved, but ranked too low 
Current guidelines are buried beneath outdated policies 
 FP3  
Factually inconsistent hallucination 
Model contradicts the retrieved information 
Fabricated compliance procedures despite the correct source documents 
 FP4  
Not in context 
Relevant chunk retrieved but filtered out 
Important policy details were removed during context optimization 
 FP5  
Not extracted 
Information in context, but not used in response 
Model ignores the retrieved compliance requirements 
 FP6  
Incorrect specificity 
Wrong level of detail for the user’s needs 
Strategic overview of when operational steps are needed 
 FP7  
Incomplete 
Partial answer when a complete response is needed 
Missing critical steps in multi-part procedures 

Use the failure-point taxonomy to choose what the suite should test, from missing content and ranking errors to unsupported answers. section 1.4

A production RAG suite can test the system at three levels:

  • Component tests isolate stages such as retrieval, reranking, and generation; run them on every code change. section 15.6.4
  • Integration tests exercise the full pipeline on an evaluation dataset before deployment, exposing interactions between stages. section 15.6.4
  • Regression tests compare the current pipeline with the previous production version on challenging or previously failed queries. Block deployment when a known-bad case worsens. section 15.6.4

Keep the test set current by adding production failures, preserving queries the system handled well, and including synthetic and expert-authored questions. Include unanswerable questions to test whether the system recognizes the limits of its knowledge base. section 14.4.2 section 15.6.4

Checking a pipeline change

After changing chunk size, run integration tests to catch effects across retrieval and generation. Then compare a curated set of reference answers to check that queries the system previously answered correctly have not regressed. section 15.6.4 section 14.2.6

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines component, integration, and regression tests with run frequencies and a deployment-blocking condition.
Examples
Uses unanswerable questions, captured production interactions, synthetic examples, expert-authored edge cases, and previously failed queries.
What it assumes
Uses an evaluation dataset as the foundation for integration and regression tests.
What you can do after
Build a living RAG test set and use it to check pipeline changes at component, integration, and regression levels.
How it connects
Connects RAG evaluation datasets with component testing, integration testing, regression testing, and ground-truth metrics.

Also explained in

Compare how 22 books explain test suite →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.