chapter fourteen

14 Evaluating RAG systems

 

This chapter covers

  • Classical retrieval metrics and why they mislead in RAG
  • The RAGAS evaluation triad: faithfulness, relevance, context
  • LLM-as-judge techniques from G-Eval through ARES
  • Generating synthetic evaluation datasets from your corpus
  • Building evaluation pipelines that diagnose failure points

Three weeks into improving a compliance assistant, a team at a mortgage lender added a cross-encoder reranker. Retrieval improved: the documents coming back were now the ones that answered the question. They shipped.

Faithfulness fell: the answers drifted away from the documents behind them. Nothing caught it. The score was lower than the week before and still above the threshold the team had set, and a threshold cannot see a trend. An engineer did, by setting last week's report beside this week's. The case study at the end of this chapter takes the regression apart claim by claim.

That failure has two halves, and this chapter is about both. The system got worse in a way its own tests were configured not to report. And the measure that moved was not the measure anyone was working on. Both problems yield to measuring differently, which for most of this chapter means asking a language model.

14.1 Classical retrieval metrics and their limits

14.1.1 The foundation: Precision and recall

14.1.2 The correlation gap: When good retrieval metrics produce bad answers

14.2 The RAGAS evaluation framework

14.2.1 Faithfulness: does the answer stick to the evidence?

14.2.2 Answer relevance: Does the answer address the question?

14.2.3 Context relevance: Did retrieval fetch what was needed?

14.2.4 Putting the triad to work

14.2.5 Limitations and caveats

14.2.6 Beyond the triad: When ground-truth still earns its keep

14.3 LLM-as-judge: From G-Eval to trained evaluators

14.3.1 G-Eval: Chain-of-thought meets evaluation

14.3.2 ARES: Trained judges with statistical guarantees

14.3.3 The bias problem: What LLM judges get wrong

14.4 Building your evaluation dataset

14.4.1 Generating balanced evaluation data from your corpus

14.4.2 Tools that do this for you

14.5 Implementing an evaluation pipeline

14.5.1 A faithfulness judge

14.5.2 Mapping scores to failure points

14.5.3 Practical considerations at scale

14.5.4 Telling a real change from noise

14.5.5 Evaluating agents and conversations

14.6 Case study: Continuous evaluation for a compliance assistant

14.7 Summary