liveBook · Manning

Evaluation

As explained in Retrieval Augmented Generation, The Foundational Ideas

Systematic evaluation is a structured process for measuring system behavior against defined criteria and test conditions.

Evaluation turns RAG reliability into an engineering practice: measure behavior against defined criteria, connect scores to failure points, and decide whether added complexity is worth its cost. section 1.1 section 1.4

Use reference-free measures to inspect different parts of the pipeline:

  • Faithfulness checks whether answer claims are supported by retrieved passages. section 1.1
  • Answer relevance checks whether the response addresses the user’s question. section 1.1
  • Context relevance checks how much retrieved material is useful. section 1.1
  • Targeted test suites expose failure points such as missing content, ranking errors, hallucinations, filtered context, and unused evidence. section 1.4

Putting a price on complexity

An agentic support assistant gained 8.23 points of retrieval accuracy but incurred 6.3 times the latency. Evaluation therefore supports a selection decision rather than assuming that a more advanced architecture is automatically better. section 1.1 section 15.1

Table 1.1 RAG failure points
Failure point
Name
 Description
Impact on financial services example
 FP1  
Missing content 
Answer doesn’t exist in the knowledge base 
Policy questions about procedures not yet documented 
 FP2  
Missed the top rank 
Correct document retrieved, but ranked too low 
Current guidelines are buried beneath outdated policies 
 FP3  
Factually inconsistent hallucination 
Model contradicts the retrieved information 
Fabricated compliance procedures despite the correct source documents 
 FP4  
Not in context 
Relevant chunk retrieved but filtered out 
Important policy details were removed during context optimization 
 FP5  
Not extracted 
Information in context, but not used in response 
Model ignores the retrieved compliance requirements 
 FP6  
Incorrect specificity 
Wrong level of detail for the user’s needs 
Strategic overview of when operational steps are needed 
 FP7  
Incomplete 
Partial answer when a complete response is needed 
Missing critical steps in multi-part procedures 

Map a poor score to the relevant RAG failure point before choosing a corrective technique or architectural change. section 1.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains three reference-free metrics and targeted failure tests, but stops short of specifying metric formulas or complete test implementations.
Examples
Uses faithfulness, answer relevance, context relevance, failure-point test suites, and an agentic assistant accuracy-latency comparison.
What you can do after
Diagnose/debug with targeted metrics and test suites, then compare accuracy gains against latency and complexity costs.
How it connects
Connects RAG evaluation to retrieval, ranking, hallucination, context optimization, agentic systems, latency, and architectural complexity.

Also explained in

Compare how 29 books explain evaluation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.