liveBook · Manning

Chapter 14: Evaluating RAG systems

What you will learn in chapter 14 of Retrieval Augmented Generation, The Foundational Ideas

The introduction shows why RAG evaluation needs separate retrieval and answer measures, while cautioning that LLM-judge scores are most useful for diagnosing and comparing systems rather than certifying quality.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction shows why RAG evaluation needs separate retrieval and answer measures, while cautioning that LLM-judge scores are most useful for diagnosing and comparing systems rather than certifying quality.

Key terms: faithfulness, answer relevance, context relevance, reference-free evaluation, LLM-as-judge, lower perplexity, gold answer, expert judgment, faithfulness score, threshold

14.1 Classical retrieval metrics and their limits

Introduces classical retrieval metrics as a starting point for RAG evaluation while warning that they were designed for a different kind of reader.

Key terms: retrieval evaluation, core metrics, concatenated context window

14.1.1 The foundation: Precision and recall

This section introduces precision, recall, F1, MRR, and NDCG, showing why a RAG retriever must be evaluated for ranking as well as relevance.

Key terms: Precision@k, Recall@k, F1 score, harmonic mean, rank position, lost in the middle, context, Mean Reciprocal Rank, MRR, first relevant result

14.1.2 The correlation gap: When good retrieval metrics produce bad answers

This section explains why query-relevance scores can misrepresent RAG quality, how eRAG evaluates documents by their usefulness to the generator, and why full-pipeline evaluation is still needed.

Key terms: relevance to the query, utility to the generator, Recall@k, relevance labels, eRAG, downstream LLM, relevance annotator, ground truth, Exact Match, F1

14.2 The RAGAS evaluation framework

Introduces RAGAS as a reference-free way to evaluate RAG outputs and use three metrics to locate pipeline failures.

Key terms: RAGAS, reference-free, ground-truth answers, faithfulness, answer relevance, context relevance, evaluation triad

14.2.1 Faithfulness: does the answer stick to the evidence?

This section explains how RAGAS measures whether a generated answer’s factual claims are supported by retrieved context.

Key terms: faithfulness, retrieved context, factually inconsistent hallucination, RAGAS, factual claims, verified claims, faithfulness score

14.2.2 Answer relevance: Does the answer address the question?

This section explains what answer relevance measures and how RAGAS estimates it without a reference answer.

Key terms: answer relevance, incorrect specificity, RAGAS, embedding similarity, cosine similarity

14.2.3 Context relevance: Did retrieval fetch what was needed?

This section defines context relevance, shows how to score retrieved text, and explains why irrelevant context matters to RAG quality and cost.

Key terms: context relevance, retrieved context, useful sentences, Insufficient Information, retrieval precision, context window

14.2.4 Putting the triad to work

This section uses the RAGAS triad to diagnose a RAG answer, then shows why the metrics must be read together and why a single score is not definitive.

Key terms: context relevance, faithfulness, answer relevance, retrieval precision, grounded, fabricated, triad, judge's own variance, confidence intervals

14.2.5 Limitations and caveats

This section explains why RAGAS scores need careful interpretation before they are used to judge a production RAG system.

Key terms: RAGAS, WikiEval, faithfulness, context relevance, prompt sensitivity, judge LLM, RAGAS scores, LLM calls, batching, caching

14.2.6 Beyond the triad: When ground-truth still earns its keep

This section explains when reference answers add value to RAG evaluation and how to combine ground-truth metrics with continuous, reference-free monitoring.

Key terms: Answer Semantic Similarity, Answer Correctness, reference answer, factual overlap, triad, ground-truth metrics, reference answers, regression suite, pipeline change

14.3 LLM-as-judge: From G-Eval to trained evaluators

Introduces LLM-based RAG evaluation as possible without reference answers while cautioning that fluent judges may not closely match expert judgment.

Key terms: RAGAS, LLMs, reference answers, expert judgment

14.3.1 G-Eval: Chain-of-thought meets evaluation

This section explains G-Eval’s criteria, step-by-step judging, and probability-weighted scoring, then adapts the pattern to judging RAG answer faithfulness.

Key terms: G-Eval, evaluation criteria, Evaluation Steps, chain-of-thought, token probabilities, probability-weighted score, rating tokens, faithfulness, G-Eval-style evaluator, geval_faithfulness

14.3.2 ARES: Trained judges with statistical guarantees

This section explains how ARES trains specialized RAG evaluators on synthetic examples and uses human annotations to put confidence intervals around their scores.

Key terms: synthetic training data, query-passage-answer triples, positive examples, negative examples, binary classifier, context relevance, prediction-powered inference, PPI, human preference validation set, correction factor

14.3.3 The bias problem: What LLM judges get wrong

This section explains why LLM judges can give biased or inconsistent RAG evaluations, why agreement between judges is insufficient validation, and when more rigorous evaluation is warranted.

Key terms: self-preference bias, position bias, style, verbosity bias, LLM judges, stochastic variance, VERA, bootstrap statistics, confidence intervals, GroUSE

14.4 Building your evaluation dataset

This section explains why a useful RAG evaluation dataset must reflect the question types users actually ask, rather than relying on convenient but skewed test questions.

Key terms: evaluation datasets, ground-truth answers, QA benchmarks, retrieval models, fact_single, summary, reasoning, hybrid search, text-weight parameter, query distribution

14.4.1 Generating balanced evaluation data from your corpus

This section shows how to generate a grounded, balanced evaluation dataset from your own corpus, including questions the supplied context cannot answer.

Key terms: statement-extraction approach, inversion, factual statements, fact_single, summary, reasoning, label, stratified analysis, unanswerable questions, FP1

14.4.2 Tools that do this for you

This section compares tools for generating synthetic RAG evaluation questions and explains why a strong test set also needs production interactions and expert-authored cases.

Key terms: Ragas, evolution, LLM-based filtering, DeepEval's Synthesizer, evolution depths, custom statement-extraction pipeline, question diversity, difficulty escalation, unanswerable questions, synthetic generation

14.5 Implementing an evaluation pipeline

This section outlines a repeatable evaluation pipeline that turns RAG test runs into diagnostic reports and recommended actions.

Key terms: balanced evaluation dataset, (query, context, response) triples, metric families, failure points, diagnostic report

14.5.1 A faithfulness judge

This section explains how a reusable faithfulness judge checks an answer claim by claim against retrieved context and reports results useful for debugging.

Key terms: faithfulness, _extract_claims, atomic factual statements, retrieved context, aggregate score, per-claim verdicts, diagnostic

14.5.2 Mapping scores to failure points

This section shows how to interpret RAG evaluation scores as diagnostic signals while recognizing what faithfulness, sufficiency, and threshold checks cannot establish.

Key terms: faithfulness, absolute threshold, calibrate, sufficient context, relevance, abstain, sufficiency, superseded policy, context relevance (CR), faithfulness (F)

14.5.3 Practical considerations at scale

This section explains how to control the cost of RAG evaluation at scale while retaining useful quality monitoring.

Key terms: faithfulness, answer relevance, context relevance, LLM call, grounding check, textual entailment, full triad, claim-decomposing judge, batching, caching

14.5.4 Telling a real change from noise

This section shows how to use uncertainty intervals and the evaluation set’s structure to decide whether a change in RAG scores is meaningful.

Key terms: automated judge, ranking of systems, single answers, passage-level correlation, resampling questions, resampling passages, interval, interval on the change, faithfulness, answer relevance

14.5.5 Evaluating agents and conversations

This section explains why evaluating agentic and multi-turn RAG requires more than scoring a single query, context, and answer.

Key terms: agentic pipeline, triad, AgentGoalAccuracy, ToolCallAccuracy, ToolCallF1, TopicAdherence, multi-turn RAG, rewrite, retrieval alone, correct passages

14.6 Case study: Continuous evaluation for a compliance assistant

A mortgage-lender case study shows how continuous RAG evaluation identifies meaningful regressions, guides fixes, and reveals weaknesses hidden by aggregate scores.

Key terms: mean, 95% interval, noise, context relevance, FP2, FP4, reranker, date-based metadata filter, faithfulness, per-claim verdicts

14.7 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 13: Context compression and pruning Chapter 15: Production RAG: Metrics, agentic systems, and continuous improvement →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.