Chapter 14: Evaluating RAG systems
The introduction shows why RAG evaluation needs separate retrieval and answer measures, while cautioning that LLM-judge scores are most useful for diagnosing and comparing systems rather than certifying quality.
After this chapter you will be able to
- The RAGAS evaluation triad. Explain the three questions in the RAGAS triad and how their scores point toward retrieval or generation problems. Describe how RAGAS can evaluate a query, context, and response without human-annotated ground truth.
- Limits of model judges. Describe documented ways an LLM judge can diverge from human judgment and distinguish a promising change to grading criteria from merely checking more claims.
- Comparative metric interpretation. Explain why a RAG evaluation score is more useful in comparisons than as a literal quality certificate, using the compliance-assistant regression as an example.
- Classical Metrics and RAG. Explain why classical retrieval metrics are useful but have limits when applied to RAG.
- Precision recall and F1. Distinguish Precision@k from Recall@k and describe how F1 combines them. Apply the section’s example to say what each score counts.
- Rank position in RAG. Explain why counting relevant retrieved documents alone can miss a distinction important to RAG answer quality.
- Position sensitive NDCG scores. Explain how NDCG@k accounts for all relevant results within the ranking and why equal Precision@k can accompany different NDCG@k scores.
- Query relevance versus answer utility. Explain why strong classical retrieval scores can coexist with poor RAG answers and why changing metric cutoffs or rank weights does not fix the underlying problem.
- Generator-based document evaluation. Describe how eRAG judges each retrieved document through the downstream LLM and why its annotations better track RAG performance than traditional relevance labels.
- Limits of retrieval-only evaluation. Distinguish what classical retrieval metrics remain useful for from failures that require examining how the generator uses retrieved context.
- Reference-free RAG evaluation. Explain why RAGAS can evaluate queries without ground-truth answers and why that matters when a knowledge base changes.
- Three diagnostic quality dimensions. Describe the three RAGAS dimensions and the kinds of pipeline failure each helps isolate.
- Faithfulness as evidence support. Explain that faithfulness asks whether each claim in a generated answer can be inferred from the retrieved context, helping detect answers that contradict their own evidence.
- Claim extraction and verification. Trace RAGAS’s two-step faithfulness evaluation: extract individual factual claims, check each against the retrieved context, and divide supported claims by total claims. Explain why this gives a more granular score than one overall judgment.
- Reverse question relevance scoring. Trace how RAGAS estimates answer relevance without a reference answer: generate plausible questions from the answer, compare each with the original question using embedding cosine similarity, and average the scores. Explain why vague or off-topic answers tend to produce less similar questions.
- Context relevance calculation. Explain that context relevance measures how much retrieved context helps answer a query, and describe how an LLM extracts useful sentences before calculating their share of all retrieved sentences.
- Score meaning and consequences. Interpret low and high context relevance scores and explain why unnecessary retrieved material can harm a RAG system.
- Triad scores and diagnosis. Explain how the three scores in the financial-services example lead to different diagnoses and follow-up actions.
- Fabrication and metric disagreement. Explain why answer relevance favored a fabricated compliance answer and why faithfulness changed the interpretation.
- Benchmark agreement limits. Explain why the reported agreement between RAGAS and human annotators may not carry over to a production corpus.
- Curated regression evaluation. Explain why a small curated set of reference answers is useful alongside continuous triad evaluation, and describe when to run each.
- Reference-free evaluation and its limits. Explain that RAGAS showed LLMs can evaluate RAG systems without reference answers, but a judge's fluent output does not guarantee close agreement with expert judgment.
- Criteria and evaluation steps. Explain how G-Eval gives an LLM an evaluation task and criteria, then prompts it to generate step-by-step evaluation procedures before scoring.
- Probability-weighted rating scores. Explain why G-Eval uses rating-token probabilities and how it turns them into a continuous score instead of relying on one generated integer.
- Synthetic examples and specialized judges. Trace how ARES creates in-domain positive and negative examples and uses them to train judges for three evaluation dimensions.
- Human-calibrated confidence intervals. Explain how prediction-powered inference combines judge predictions with a small human-annotated sample to report uncertainty around system-quality scores.
- Changing biases in LLM judges. Describe how self-preference, answer order, formatting, and answer length can affect an LLM judge, and explain why a team should test its own judge rather than assume an older bias finding still applies.
- Score variance and repeated evaluation. Explain how the same input can receive different scores across runs and how VERA uses repeated evaluation samples and bootstrap confidence intervals to account for that randomness.
- Judge agreement versus correctness. Explain why high correlation with a stronger judge does not establish that a cheaper judge catches grounded-answer failures, using GroUSE’s finding as evidence.
- Question type distribution skew. Explain how an excess of fact_single questions can make RAG evaluation and tuning misleading. Describe the contrast with underrepresented reasoning questions and why retrieval settings may need to differ by question type.
- Grounded answer-first generation. Explain how extracting factual statements from a passage and then turning those statements into questions keeps synthetic question–answer pairs grounded in the corpus.
- Balanced question-type coverage. Describe how the pipeline uses corpus passages to produce fact_single, summary, and reasoning questions, and how labels enable scores to be broken down by type.
- Synthetic question generation approaches. Distinguish how Ragas and DeepEval generate challenging evaluation questions. Describe the benefit and cost of Ragas’s multi-step approach.
- Balanced synthetic test sets. Justify combining custom statement extraction with a framework tool, and explain why unanswerable questions belong in the test set.
- Production and expert examples. Explain how successful production interactions and expert-authored questions complement synthetic evaluation data.
- Four-phase evaluation pipeline. Trace how the pipeline moves from a document corpus to a diagnostic report: generate a balanced test dataset, run queries to collect query–context–response triples, score the triples, and map aggregate scores to failure points and recommended actions. Explain why the pipeline is automated and run repeatedly.
- Claim by claim verification. Trace the judge’s two-step process: an LLM extracts independent factual claims from the answer, and each claim is checked against the retrieved context.
- Diagnostic per claim verdicts. Explain why the evaluate method returns per-claim verdicts alongside an aggregate faithfulness score.
- Implementation-Dependent Score Thresholds. Explain why faithfulness scores are more useful for comparing configurations than as universal measures of answer quality, and describe how to calibrate a threshold.
- Context Sufficiency and Abstention. Distinguish relevant context from sufficient context, and explain why a sufficiency check can help a system decide when to abstain.
- Superseded Evidence Blind Spot. Explain why an answer can pass faithfulness, relevance, and sufficiency checks yet still be wrong when retrieved evidence is superseded.
- Score-Pattern Failure Diagnosis. Describe how the mapper uses context relevance, faithfulness, and answer relevance scores to identify likely failure points and actions.
- Tiered grounding checks. Describe why a small grounding checker can run on every answer, with flagged answers escalated to the full triad and a claim-decomposing judge.
- Passage grouped resampling. Explain why confidence intervals for this evaluation should resample passages with their questions intact rather than resample individual questions.
- Changes against uncertainty intervals. Apply the reported intervals to distinguish the faithfulness regression from the inconclusive answer-relevance change, and explain why smaller differences need larger evaluation sets.
- Agentic path evaluation. Explain why a correct final answer can hide wasteful or unnecessary agent steps, and describe what path-level metrics measure and what limits their use.
- Multi-turn failure diagnosis. Describe why conversational RAG can fail at query rewriting, retrieval, or generation, and explain how comparing generation with correct versus retrieved passages helps isolate retrieval's effect.
- Retrieval failure and repair. Trace how the team's retrieval diagnosis led to a reranker and date-based filter, and describe the resulting context-relevance gain.
- Faithfulness regression after retrieval. Explain why better retrieval exposed a faithfulness problem and how the team addressed it.
- Question-type performance gap. Explain what the balanced question-type breakdown revealed that the aggregate scores concealed.
- Continuous evaluation flywheel. Describe how the fixed synthetic set and production-query sample work together in the team's weekly measure, diagnose, fix, and remeasure loop.
What it builds on
- The RAG model architecture (chapter 4) · Introduces the retriever and generator as distinct RAG components.
- The retriever (chapter 4) · Assumes the role of the retriever in supplying documents to RAG.
- The generator (chapter 4) · Assumes retrieved context is used by the generator.
- The hallucination challenge (chapter 1) · Introduces the hallucination challenge that faithfulness helps detect.
- The costs of long context (chapter 13) · Discusses the costs of long context.
- What the tokens cannot see (chapter 9) · The earlier example of reflection tokens accepting a superseded policy.
Section by section
Introduction
The introduction shows why RAG evaluation needs separate retrieval and answer measures, while cautioning that LLM-judge scores are most useful for diagnosing and comparing systems rather than certifying quality.
- The RAGAS evaluation triad: Explain the three questions in the RAGAS triad and how their scores point toward retrieval or generation problems. Describe how RAGAS can evaluate a query, context, and response without human-annotated ground truth.
Why it matters: Separate scores can show which part of a RAG system needs investigation instead of hiding the problem in one overall measure. - Limits of model judges: Describe documented ways an LLM judge can diverge from human judgment and distinguish a promising change to grading criteria from merely checking more claims.
Why it matters: A judge's convenient score can reflect its biases rather than the quality an expert would see. - Comparative metric interpretation: Explain why a RAG evaluation score is more useful in comparisons than as a literal quality certificate, using the compliance-assistant regression as an example.
Why it matters: Comparisons can catch regressions even when an individual score still passes a test.
Key terms: faithfulness, answer relevance, context relevance, reference-free evaluation, LLM-as-judge, lower perplexity, gold answer, expert judgment, faithfulness score, threshold
14.1 Classical retrieval metrics and their limits
Introduces classical retrieval metrics as a starting point for RAG evaluation while warning that they were designed for a different kind of reader.
- Classical Metrics and RAG: Explain why classical retrieval metrics are useful but have limits when applied to RAG.
Why it matters: The difference in who consumes the retrieved material explains why familiar retrieval metrics need scrutiny in RAG evaluation.
Key terms: retrieval evaluation, core metrics, concatenated context window
14.1.1 The foundation: Precision and recall
This section introduces precision, recall, F1, MRR, and NDCG, showing why a RAG retriever must be evaluated for ranking as well as relevance.
- Precision recall and F1: Distinguish Precision@k from Recall@k and describe how F1 combines them. Apply the section’s example to say what each score counts.
Why it matters: These scores establish what retrieval found before you examine where it placed the relevant documents. - Rank position in RAG: Explain why counting relevant retrieved documents alone can miss a distinction important to RAG answer quality.
Why it matters: Two retrievers can find the same relevant documents yet give the generator differently useful context. - First relevant result ranking: Describe what Mean Reciprocal Rank measures and how the section averages reciprocal first-relevant ranks across queries.
Why it matters: MRR makes the earliest relevant result visible in evaluation. - Position sensitive NDCG scores: Explain how NDCG@k accounts for all relevant results within the ranking and why equal Precision@k can accompany different NDCG@k scores.
Why it matters: NDCG distinguishes a retriever that surfaces useful evidence early from one that buries it.
Key terms: Precision@k, Recall@k, F1 score, harmonic mean, rank position, lost in the middle, context, Mean Reciprocal Rank, MRR, first relevant result
14.1.2 The correlation gap: When good retrieval metrics produce bad answers
This section explains why query-relevance scores can misrepresent RAG quality, how eRAG evaluates documents by their usefulness to the generator, and why full-pipeline evaluation is still needed.
- Query relevance versus answer utility: Explain why strong classical retrieval scores can coexist with poor RAG answers and why changing metric cutoffs or rank weights does not fix the underlying problem.
Why it matters: A healthy retrieval score does not by itself show that the retrieved evidence can support a correct answer. - Generator-based document evaluation: Describe how eRAG judges each retrieved document through the downstream LLM and why its annotations better track RAG performance than traditional relevance labels.
Why it matters: Testing what a document helps the generator answer makes retrieval evaluation more closely reflect the task the RAG system must perform. - Limits of retrieval-only evaluation: Distinguish what classical retrieval metrics remain useful for from failures that require examining how the generator uses retrieved context.
Why it matters: Checking retrieval alone can miss failures that occur after useful evidence reaches the prompt.
Key terms: relevance to the query, utility to the generator, Recall@k, relevance labels, eRAG, downstream LLM, relevance annotator, ground truth, Exact Match, F1
14.2 The RAGAS evaluation framework
Introduces RAGAS as a reference-free way to evaluate RAG outputs and use three metrics to locate pipeline failures.
- Reference-free RAG evaluation: Explain why RAGAS can evaluate queries without ground-truth answers and why that matters when a knowledge base changes.
Why it matters: Reference-free evaluation makes it practical to assess queries even as the underlying documents change. - Three diagnostic quality dimensions: Describe the three RAGAS dimensions and the kinds of pipeline failure each helps isolate.
Why it matters: The separate dimensions help identify which part of a RAG pipeline needs attention.
Key terms: RAGAS, reference-free, ground-truth answers, faithfulness, answer relevance, context relevance, evaluation triad
14.2.1 Faithfulness: does the answer stick to the evidence?
This section explains how RAGAS measures whether a generated answer’s factual claims are supported by retrieved context.
- Faithfulness as evidence support: Explain that faithfulness asks whether each claim in a generated answer can be inferred from the retrieved context, helping detect answers that contradict their own evidence.
Why it matters: This check can catch an answer that sounds convincing but conflicts with the material it retrieved. - Claim extraction and verification: Trace RAGAS’s two-step faithfulness evaluation: extract individual factual claims, check each against the retrieved context, and divide supported claims by total claims. Explain why this gives a more granular score than one overall judgment.
Why it matters: Checking claims individually shows how much of an answer is supported instead of reducing mixed answers to a vague overall verdict.
Key terms: faithfulness, retrieved context, factually inconsistent hallucination, RAGAS, factual claims, verified claims, faithfulness score
14.2.2 Answer relevance: Does the answer address the question?
This section explains what answer relevance measures and how RAGAS estimates it without a reference answer.
- Answer relevance criterion: Explain that answer relevance asks whether a response addresses the user's question, rather than merely discussing the same broad topic. Describe how incomplete, off-topic, padded, or wrongly specific answers can score poorly.
Why it matters: An answer can sound related to a topic yet fail to answer the user's actual question. - Reverse question relevance scoring: Trace how RAGAS estimates answer relevance without a reference answer: generate plausible questions from the answer, compare each with the original question using embedding cosine similarity, and average the scores. Explain why vague or off-topic answers tend to produce less similar questions.
Why it matters: This explains how the metric can assess whether an answer fits a question without needing a model answer.
Key terms: answer relevance, incorrect specificity, RAGAS, embedding similarity, cosine similarity
14.2.3 Context relevance: Did retrieval fetch what was needed?
This section defines context relevance, shows how to score retrieved text, and explains why irrelevant context matters to RAG quality and cost.
- Context relevance calculation: Explain that context relevance measures how much retrieved context helps answer a query, and describe how an LLM extracts useful sentences before calculating their share of all retrieved sentences.
Why it matters: The score asks how much retrieved text can help the generator, rather than merely whether a relevant document was found. - Score meaning and consequences: Interpret low and high context relevance scores and explain why unnecessary retrieved material can harm a RAG system.
Why it matters: A retriever can return useful evidence yet still burden the generator with too much irrelevant text.
Key terms: context relevance, retrieved context, useful sentences, Insufficient Information, retrieval precision, context window
14.2.4 Putting the triad to work
This section uses the RAGAS triad to diagnose a RAG answer, then shows why the metrics must be read together and why a single score is not definitive.
- Triad scores and diagnosis: Explain how the three scores in the financial-services example lead to different diagnoses and follow-up actions.
Why it matters: The three scores help locate different problems rather than reducing answer quality to one number. - Fabrication and metric disagreement: Explain why answer relevance favored a fabricated compliance answer and why faithfulness changed the interpretation.
Why it matters: An on-topic answer can still be a confident fabrication. - Single-run judge variance: Describe why a score from one evaluation run on one sample should not be treated as a definitive measurement.
Why it matters: A changed score may reflect judge variability rather than a changed system.
Key terms: context relevance, faithfulness, answer relevance, retrieval precision, grounded, fabricated, triad, judge's own variance, confidence intervals
14.2.5 Limitations and caveats
This section explains why RAGAS scores need careful interpretation before they are used to judge a production RAG system.
- Benchmark agreement limits: Explain why the reported agreement between RAGAS and human annotators may not carry over to a production corpus.
Why it matters: A strong benchmark result does not by itself show that an evaluator will catch subtle errors in your system. - Judge prompt and model sensitivity: Describe how the judge's prompt following, model choice, and run-to-run variation can affect RAGAS scores.
Why it matters: A score change can reflect a change in the judge rather than a change in the RAG system. - Evaluation call costs: Explain why running RAGAS over a large evaluation set can be costly and how that affects continuous monitoring.
Why it matters: Evaluation frequency and dataset size have a real operating cost. - Human disagreement noise floor: Explain why a small difference between evaluation scores may not indicate a real quality change.
Why it matters: Treating every small score movement as an improvement risks drawing conclusions from noise. - Generator judge self preference: Describe the circular evaluation risk when the generator and judge come from the same model family, and the stated tradeoff of using a different family.
Why it matters: Judge selection can affect both the credibility and cost of an evaluation.
Key terms: RAGAS, WikiEval, faithfulness, context relevance, prompt sensitivity, judge LLM, RAGAS scores, LLM calls, batching, caching
14.2.6 Beyond the triad: When ground-truth still earns its keep
This section explains when reference answers add value to RAG evaluation and how to combine ground-truth metrics with continuous, reference-free monitoring.
- Reference-based answer metrics: Distinguish Answer Semantic Similarity from Answer Correctness and explain why both require a reference answer.
Why it matters: These metrics let you compare an answer with a curated target rather than relying only on reference-free scores. - Curated regression evaluation: Explain why a small curated set of reference answers is useful alongside continuous triad evaluation, and describe when to run each.
Why it matters: A stable regression set can reveal a loss on previously successful queries that broad monitoring might obscure.
Key terms: Answer Semantic Similarity, Answer Correctness, reference answer, factual overlap, triad, ground-truth metrics, reference answers, regression suite, pipeline change
14.3 LLM-as-judge: From G-Eval to trained evaluators
Introduces LLM-based RAG evaluation as possible without reference answers while cautioning that fluent judges may not closely match expert judgment.
- Reference-free evaluation and its limits: Explain that RAGAS showed LLMs can evaluate RAG systems without reference answers, but a judge's fluent output does not guarantee close agreement with expert judgment.
Why it matters: This distinction helps you treat an LLM judge's verdict as an evaluation signal rather than unquestioned expert judgment.
Key terms: RAGAS, LLMs, reference answers, expert judgment
14.3.1 G-Eval: Chain-of-thought meets evaluation
This section explains G-Eval’s criteria, step-by-step judging, and probability-weighted scoring, then adapts the pattern to judging RAG answer faithfulness.
- Criteria and evaluation steps: Explain how G-Eval gives an LLM an evaluation task and criteria, then prompts it to generate step-by-step evaluation procedures before scoring.
Why it matters: Explicit criteria and evaluation steps make the judge’s assessment more structured than a bare request for a rating. - Probability-weighted rating scores: Explain why G-Eval uses rating-token probabilities and how it turns them into a continuous score instead of relying on one generated integer.
Why it matters: A continuous score can distinguish outputs that would otherwise receive the same integer rating. - RAG faithfulness judge adaptation: Describe how the section’s G-Eval-style example judges RAG answer faithfulness, including its simpler scoring choice when token probabilities are unavailable.
Why it matters: The example shows how to use the judging pattern even when an LLM API cannot provide token probabilities.
Key terms: G-Eval, evaluation criteria, Evaluation Steps, chain-of-thought, token probabilities, probability-weighted score, rating tokens, faithfulness, G-Eval-style evaluator, geval_faithfulness
14.3.2 ARES: Trained judges with statistical guarantees
This section explains how ARES trains specialized RAG evaluators on synthetic examples and uses human annotations to put confidence intervals around their scores.
- Synthetic examples and specialized judges: Trace how ARES creates in-domain positive and negative examples and uses them to train judges for three evaluation dimensions.
Why it matters: Domain-specific examples give the judges a training signal for scoring RAG outputs at scale. - Human-calibrated confidence intervals: Explain how prediction-powered inference combines judge predictions with a small human-annotated sample to report uncertainty around system-quality scores.
Why it matters: The interval helps distinguish a meaningful quality change from an uncertain score. - Reported judge performance: Describe what the reported comparisons indicate about ARES judges' accuracy and behavior under domain shifts.
Why it matters: These results provide evidence for the section's claim that trained judges can improve on prompted evaluation.
Key terms: synthetic training data, query-passage-answer triples, positive examples, negative examples, binary classifier, context relevance, prediction-powered inference, PPI, human preference validation set, correction factor
14.3.3 The bias problem: What LLM judges get wrong
This section explains why LLM judges can give biased or inconsistent RAG evaluations, why agreement between judges is insufficient validation, and when more rigorous evaluation is warranted.
- Changing biases in LLM judges: Describe how self-preference, answer order, formatting, and answer length can affect an LLM judge, and explain why a team should test its own judge rather than assume an older bias finding still applies.
Why it matters: A score may reflect a judge’s preferences about presentation rather than the quality you meant to measure. - Score variance and repeated evaluation: Explain how the same input can receive different scores across runs and how VERA uses repeated evaluation samples and bootstrap confidence intervals to account for that randomness.
Why it matters: Repeated scoring helps distinguish a meaningful evaluation result from a judge’s run-to-run randomness. - Judge agreement versus correctness: Explain why high correlation with a stronger judge does not establish that a cheaper judge catches grounded-answer failures, using GroUSE’s finding as evidence.
Why it matters: A cheaper judge can reproduce another model’s scores while still missing important RAG failures. - Evaluation rigor by deployment stage: Distinguish the section’s recommendation for teams starting RAG evaluation from its recommendation for production teams where regressions have business consequences.
Why it matters: Evaluation effort should match the consequences of missing a regression or raising a false alarm.
Key terms: self-preference bias, position bias, style, verbosity bias, LLM judges, stochastic variance, VERA, bootstrap statistics, confidence intervals, GroUSE
14.4 Building your evaluation dataset
This section explains why a useful RAG evaluation dataset must reflect the question types users actually ask, rather than relying on convenient but skewed test questions.
- Benchmark suitability for RAG: Distinguish the value of standard QA benchmarks for comparing retrieval models in isolation from their limitations when evaluating an enterprise RAG system. Explain why a few demo queries or a large set of manually written questions may also give an incomplete test.
Why it matters: The questions you choose determine which system failures your evaluation can reveal. - Question type distribution skew: Explain how an excess of fact_single questions can make RAG evaluation and tuning misleading. Describe the contrast with underrepresented reasoning questions and why retrieval settings may need to differ by question type.
Why it matters: An unbalanced test set can reward improvements that do not help with the questions users actually bring.
Key terms: evaluation datasets, ground-truth answers, QA benchmarks, retrieval models, fact_single, summary, reasoning, hybrid search, text-weight parameter, query distribution
14.4.1 Generating balanced evaluation data from your corpus
This section shows how to generate a grounded, balanced evaluation dataset from your own corpus, including questions the supplied context cannot answer.
- Grounded answer-first generation: Explain how extracting factual statements from a passage and then turning those statements into questions keeps synthetic question–answer pairs grounded in the corpus.
Why it matters: An evaluation set cannot reliably test a RAG system if its own answers are fabricated. - Balanced question-type coverage: Describe how the pipeline uses corpus passages to produce fact_single, summary, and reasoning questions, and how labels enable scores to be broken down by type.
Why it matters: A type-labeled dataset can reveal which kinds of questions a system struggles with instead of hiding them in one overall score. - Unanswerable question construction: Describe how to pair questions generated from unrelated corpus passages with an original passage to test whether the system recognizes insufficient information.
Why it matters: Evaluation should test whether a system admits that the provided context cannot support an answer.
Key terms: statement-extraction approach, inversion, factual statements, fact_single, summary, reasoning, label, stratified analysis, unanswerable questions, FP1
14.4.2 Tools that do this for you
This section compares tools for generating synthetic RAG evaluation questions and explains why a strong test set also needs production interactions and expert-authored cases.
- Synthetic question generation approaches: Distinguish how Ragas and DeepEval generate challenging evaluation questions. Describe the benefit and cost of Ragas’s multi-step approach.
Why it matters: The tools offer different ways to add realistic difficulty to an evaluation set without manually writing every question. - Balanced synthetic test sets: Justify combining custom statement extraction with a framework tool, and explain why unanswerable questions belong in the test set.
Why it matters: A varied test set must check both domain-specific performance and whether the system handles questions it cannot answer. - Production and expert examples: Explain how successful production interactions and expert-authored questions complement synthetic evaluation data.
Why it matters: Broad synthetic coverage alone can miss both regressions in real user queries and domain-specific edge cases.
Key terms: Ragas, evolution, LLM-based filtering, DeepEval's Synthesizer, evolution depths, custom statement-extraction pipeline, question diversity, difficulty escalation, unanswerable questions, synthetic generation
14.5 Implementing an evaluation pipeline
This section outlines a repeatable evaluation pipeline that turns RAG test runs into diagnostic reports and recommended actions.
- Four-phase evaluation pipeline: Trace how the pipeline moves from a document corpus to a diagnostic report: generate a balanced test dataset, run queries to collect query–context–response triples, score the triples, and map aggregate scores to failure points and recommended actions. Explain why the pipeline is automated and run repeatedly.
Why it matters: The pipeline makes evaluation useful for finding what to fix, rather than merely producing scores.
Key terms: balanced evaluation dataset, (query, context, response) triples, metric families, failure points, diagnostic report
14.5.1 A faithfulness judge
This section explains how a reusable faithfulness judge checks an answer claim by claim against retrieved context and reports results useful for debugging.
- Claim by claim verification: Trace the judge’s two-step process: an LLM extracts independent factual claims from the answer, and each claim is checked against the retrieved context.
Why it matters: Checking claims separately makes it possible to identify exactly which parts of an answer lack support. - Diagnostic per claim verdicts: Explain why the evaluate method returns per-claim verdicts alongside an aggregate faithfulness score.
Why it matters: The verdicts turn a low metric into a clue about what the system fabricated.
Key terms: faithfulness, _extract_claims, atomic factual statements, retrieved context, aggregate score, per-claim verdicts, diagnostic
14.5.2 Mapping scores to failure points
This section shows how to interpret RAG evaluation scores as diagnostic signals while recognizing what faithfulness, sufficiency, and threshold checks cannot establish.
- Implementation-Dependent Score Thresholds: Explain why faithfulness scores are more useful for comparing configurations than as universal measures of answer quality, and describe how to calibrate a threshold.
Why it matters: A threshold that looks authoritative can mislead you unless it fits the judge and the quality standard you actually use. - Context Sufficiency and Abstention: Distinguish relevant context from sufficient context, and explain why a sufficiency check can help a system decide when to abstain.
Why it matters: A model may produce a confident answer from passages that look relevant but do not actually contain what the question requires. - Superseded Evidence Blind Spot: Explain why an answer can pass faithfulness, relevance, and sufficiency checks yet still be wrong when retrieved evidence is superseded.
Why it matters: Even strong evaluation scores cannot establish that the evidence itself is up to date. - Score-Pattern Failure Diagnosis: Describe how the mapper uses context relevance, faithfulness, and answer relevance scores to identify likely failure points and actions.
Why it matters: The pattern of scores helps direct investigation toward retrieval or generation rather than treating every low score as the same problem.
Key terms: faithfulness, absolute threshold, calibrate, sufficient context, relevance, abstain, sufficiency, superseded policy, context relevance (CR), faithfulness (F)
14.5.3 Practical considerations at scale
This section explains how to control the cost of RAG evaluation at scale while retaining useful quality monitoring.
- Evaluation call costs: Explain why running the full evaluation pipeline on many samples becomes costly, using the calls required by faithfulness, answer relevance, and context relevance.
Why it matters: Knowing where the calls come from helps you choose where to spend an evaluation budget. - Tiered grounding checks: Describe why a small grounding checker can run on every answer, with flagged answers escalated to the full triad and a claim-decomposing judge.
Why it matters: A cheap first pass can broaden monitoring without paying for the full evaluation on every answer. - Batching caching and result storage: Describe how batching calls, caching repeated work, and storing structured results reduce evaluation overhead.
Why it matters: These practices make repeated evaluations less wasteful and their results easier to compare. - Development and production evaluation cadence: Distinguish when to run full evaluations during development from how to sample and alert during production monitoring.
Why it matters: Different cadences let you test changes thoroughly while monitoring live quality at manageable cost.
Key terms: faithfulness, answer relevance, context relevance, LLM call, grounding check, textual entailment, full triad, claim-decomposing judge, batching, caching
14.5.4 Telling a real change from noise
This section shows how to use uncertainty intervals and the evaluation set’s structure to decide whether a change in RAG scores is meaningful.
- Automated judge score limits: Explain why an automated judge’s score is more useful for comparing systems than certifying the quality of a single answer.
Why it matters: It keeps an evaluation score from being mistaken for proof that a particular answer is good. - Passage grouped resampling: Explain why confidence intervals for this evaluation should resample passages with their questions intact rather than resample individual questions.
Why it matters: Preserving shared passage effects prevents an apparently precise score from overstating the evidence. - Changes against uncertainty intervals: Apply the reported intervals to distinguish the faithfulness regression from the inconclusive answer-relevance change, and explain why smaller differences need larger evaluation sets.
Why it matters: It helps the team act on a supported regression without treating a small, noisy movement as a real change. - Small sample interval choice: Distinguish the suggested interval methods for a small pass-or-fail evaluation and for the grouped faithfulness scores discussed here.
Why it matters: The choice of interval method affects how confidently a team can interpret an evaluation result.
Key terms: automated judge, ranking of systems, single answers, passage-level correlation, resampling questions, resampling passages, interval, interval on the change, faithfulness, answer relevance
14.5.5 Evaluating agents and conversations
This section explains why evaluating agentic and multi-turn RAG requires more than scoring a single query, context, and answer.
- Agentic path evaluation: Explain why a correct final answer can hide wasteful or unnecessary agent steps, and describe what path-level metrics measure and what limits their use.
Why it matters: A good answer alone does not tell you whether an agent took an appropriate path to it. - Multi-turn failure diagnosis: Describe why conversational RAG can fail at query rewriting, retrieval, or generation, and explain how comparing generation with correct versus retrieved passages helps isolate retrieval's effect.
Why it matters: Separating retrieval from generation helps locate the cause of a conversational failure.
Key terms: agentic pipeline, triad, AgentGoalAccuracy, ToolCallAccuracy, ToolCallF1, TopicAdherence, multi-turn RAG, rewrite, retrieval alone, correct passages
14.6 Case study: Continuous evaluation for a compliance assistant
A mortgage-lender case study shows how continuous RAG evaluation identifies meaningful regressions, guides fixes, and reveals weaknesses hidden by aggregate scores.
- Meaningful score changes: Distinguish changes that exceed the evaluation's noise from small score movements that should be treated as flat.
Why it matters: The team needs to separate real changes from noise before acting on weekly scores. - Retrieval failure and repair: Trace how the team's retrieval diagnosis led to a reranker and date-based filter, and describe the resulting context-relevance gain.
Why it matters: A relevant-looking passage can still be the wrong evidence for a current compliance question. - Faithfulness regression after retrieval: Explain why better retrieval exposed a faithfulness problem and how the team addressed it.
Why it matters: Improving one RAG component can expose a separate weakness rather than improve every score at once. - Question-type performance gap: Explain what the balanced question-type breakdown revealed that the aggregate scores concealed.
Why it matters: A high overall score can hide poor performance on questions users need the assistant to answer. - Continuous evaluation flywheel: Describe how the fixed synthetic set and production-query sample work together in the team's weekly measure, diagnose, fix, and remeasure loop.
Why it matters: Combining repeatable tests with live-query coverage lets evaluation drive ongoing improvement rather than end at deployment.
Key terms: mean, 95% interval, noise, context relevance, FP2, FP4, reranker, date-based metadata filter, faithfulness, per-claim verdicts
14.7 Summary
Where this is used later
- (chapter intro) (chapter 15) · The gap between retrieval metrics and answer quality.
- RankRAG: Unifying ranking and generation (chapter 15) · Introduces the gap between retrieval relevance and answer quality.
- The testing pyramid for RAG (chapter 15) · the evaluation dataset that this section uses as a testing foundation
- Complexity versus performance (chapter 15) · Using an evaluation pipeline to measure whether requirements are met.
- Version your pipeline as a unit (chapter 15) · The evaluation pipeline used to test changes.
- Building the feedback loop (chapter 15) · Mapping evaluation scores to failure points.
- When agentic RAG helps and when it does not (chapter 15) · Provides evaluation criteria for judging when an agentic approach helps.
Concepts explained in this chapter
- RagasRAGAS is a framework for evaluating retrieval-augmented generation systems with metrics including faithfulness, answer relevance, and context relevance.
- fidelityFaithfulness measures whether a generated response is supported by the retrieved context rather than introducing unsupported claims.
- relevanceAnswer relevance is a metric measuring how well a generated answer addresses the user's question.
- context relevanceContext relevance is a metric measuring how much retrieved context is useful for answering the question.
- evaluation datasetAn evaluation set is a collection of examples used to measure system performance under specified evaluation conditions.
- reference-free evaluationReference-free RAG evaluation assesses retrieval-augmented generation outputs without requiring reference answers.
- reasoning questionA reasoning question or query requires interpretation, comparison, inference, or logical combination beyond direct extraction of a stated fact.
- confidence intervalA 95% confidence interval is an estimated range constructed by a statistical procedure intended to contain the relevant population quantity with 95% confidence.
- LLM-as-a-judgeAn LLM judge is a large language model configured to evaluate the quality or correctness of generated outputs.
- G-EvalG-Eval is an LLM-based evaluation technique that generates evaluation steps and assigns probability-weighted scores to outputs.
- synthetic dataSynthetic data is artificially generated data produced to resemble or serve the role of real data.
- evaluation triadThe evaluation triad is a three-dimensional evaluation framework comprising faithfulness, answer relevance, and context relevance.
- component-wise evaluationComponent-wise evaluation maps separate evaluation metrics to the system components responsible for the measured behavior.
- failure point mappingFailure point mapping links evaluation dimensions to specific categories of system failure.
- metrics-driven developmentMetrics-Driven Development is an iterative development approach that uses evaluation metric scores to guide system changes.
- DeepEvalDeepEval is a software framework for evaluating large language model applications and generating evaluation data.
- eRAGeRAG is a retrieval-augmented generation evaluation method that uses per-document language-model judgments to estimate actual system performance.
- fact-single questionA fact_single question is an evaluation question whose answer can be obtained as a single factual item from context.
- continuous improvementA closed-loop system observes an intermediate or resulting state and uses that evaluation to decide subsequent actions.
- groundingGrounding is conditioning a generated answer on retrieved evidence so that its claims can be checked against supporting passages.
- ARESARES is an automated evaluation framework that trains specialized judges and applies statistical inference to evaluate retrieval-augmented generation systems.
- sufficient contextSufficient context is the condition in which the retrieved set collectively contains the information needed to answer the query.
- unanswerable questionAn unanswerable question is an evaluation question whose answer is not supported by the system's knowledge base or supplied context.
- chain-of-thought promptingChain-of-thought reasoning is a reasoning technique that generates or follows intermediate steps between a problem and its answer.
- probability theoryProbability is a numerical measure representing the likelihood of an event or outcome.
- system promptA system prompt is a high-priority instruction context that establishes a language model's behavior and constraints.
- test suiteA test suite is a collection of tests designed to assess specified behaviors or failure conditions in a system.
- recallRecall is the proportion of relevant items that a retrieval or classification system successfully returns.
- Precision@kPrecision@k is the proportion of the top k retrieved results that are relevant to the query.
- alpha-ndcgα-nDCG is an intent-aware ranking metric that rewards relevant coverage of multiple query facets while reducing credit for redundancy.
- mean reciprocal rankMean reciprocal rank is the mean of the reciprocal ranks assigned to the first correct result across evaluation queries.
- precisionPrecision is the proportion of retrieved items that are relevant according to the evaluation criterion.
- position biasPosition bias is an evaluation bias in which a judge favors whichever option appears first in a comparison.
- cachingCaching stores reusable computation results or retrieved information to avoid repeating expensive operations.
- benchmarkA benchmark is a standardized evaluation task, dataset, or reference result used to compare system performance.
- hybrid searchHybrid search combines dense and sparse retrieval methods to improve the range and quality of retrieved results.
- regressionA regression is a deterioration in system behavior caused by a change that previously improved or preserved performance.
- f-scoreF1 score is the harmonic mean of precision and recall.
- probability-weighted scoringProbability-weighted scoring aggregates evaluation judgments according to the model's assigned outcome probabilities.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
