Chapter 8: RAG application evaluation

Introduction

This chapter introduces a benchmark-based approach to evaluating a RAG agent at tool-selection, retrieval, answer-generation, and end-to-end levels using RAGAS.

• Benchmark evaluation purpose: Explain why a RAG application needs a carefully constructed benchmark dataset and how benchmarking both measures an agent’s capabilities and defines its intended scope.
  Why it matters: A benchmark turns vague impressions of agent quality into a repeatable way to measure and improve it.
• Pipeline evaluation stages: Describe the distinct stages at which a RAG pipeline can be evaluated: retrieval-tool selection, retrieved-context relevance and sufficiency, answer generation from context, and the pipeline as a whole.
  Why it matters: Separating stages lets you locate whether a bad answer came from routing, retrieval, or generation.
• Retrieval tool fit: Distinguish the retrieval roles described in the chapter: vector search retrieves semantically relevant documents, Cypher templates support precise structured database queries, and text2cypher enables dynamic, flexible graph-based queries.
  Why it matters: The agent can be evaluated meaningfully only when its chosen retrieval tool is judged against the needs of the question.

Key terms: benchmark evaluation, benchmark dataset, benchmark questions, agent capabilities, tool selection, retrieval relevance, retrieved context, contextually relevant answers, end-to-end evaluation, vector search

8.1 Designing the benchmark dataset

This section explains how to design a diverse benchmark dataset that tests the decision-making, retrieval, response, and usability capabilities of a RAG agent.

• Benchmark coverage strategy: Describe how a benchmark dataset should use diverse queries to test the important components and behaviors of a RAG pipeline rather than testing only straightforward answer generation.
  Why it matters: A broad benchmark reveals where an agent fails under realistic conditions and supports targeted improvements before deployment.
• Retrieval decision test cases: Explain that benchmark queries should test tool selection, entity and value mapping, and multistep retrieval, including cases where results from one retrieval must drive a subsequent retrieval.
  Why it matters: These tests determine whether the agent can obtain the right information through the retrieval decisions required by complex user questions.
• Usability and ambiguity cases: Describe the need to include ambiguous queries, long-tail concepts, situations with multiple applicable retrieval methods, and conversational interactions such as greetings, clarification, and capability communication.
  Why it matters: A RAG agent must remain useful and understandable when users are unclear, unusual, or conversational rather than perfectly precise.
• Targeted robustness improvement: Explain that systematic benchmarking across these conditions reveals how well the agent performs and enables targeted changes that improve robustness and reliability in real-world deployment.
  Why it matters: The point of evaluation is not merely to score the system, but to identify specific improvements that make it more dependable.

Key terms: benchmark dataset, input queries, RAG pipeline, decision making, response generation, functional coverage, tool selection evaluation, retrieval method, entity and value mapping, multistep retrieval scenarios

8.1.1 Coming up with test examples

This section shows how to build a diverse end-to-end benchmark dataset of questions and ground truths that exercises all of a RAG agent’s capabilities.

• End-to-end benchmark examples: Describe a benchmark test example as a user question paired with a corresponding ground truth response, and explain that such examples enable reliable end-to-end assessment of system output.
  Why it matters: A benchmark can only assess an agent consistently when each test question has a defined expected result.
• Dynamic Cypher ground truth: Explain how a Cypher query can define ground truth dynamically instead of storing a static expected-answer string, and why this keeps graph-database benchmark cases valid when underlying data changes.
  Why it matters: Query-based ground truth avoids manually rewriting expected answers whenever the graph’s data changes.
• Functional coverage categories: Describe the range of benchmark examples needed to cover the agent’s functionality: greetings, user guidance, irrelevant questions, dedicated-tool questions, value mapping, text2cypher questions, and in-domain questions with missing data.
  Why it matters: Diverse test cases reveal whether an agent handles its full scope rather than only straightforward database questions.
• Static conversational responses: Explain that greetings, guidance requests, and irrelevant questions can use simple RETURN Cypher statements to specify static responses without looking up information in the database, including the agent’s movie-and-casts scope.
  Why it matters: Testing scope-setting responses verifies that the agent can guide users and decline unrelated requests appropriately.
• Tool selection and value mapping: Describe how benchmark questions can test use of the movie-by-actor and actor-by-movie tools as well as mapping a user’s movie or actor wording to the correct database value; explain why dedicated mapping is especially important for lesser-known or private data.
  Why it matters: Correct tool use and entity mapping determine whether an agent can retrieve the intended graph records accurately.
• Text2cypher query coverage: Explain why aggregation, filtering, and relationship questions belong in the benchmark: when no dedicated tool supports them, the LLM must use text2cypher to construct an appropriate Cypher statement from the graph schema.
  Why it matters: These cases test whether the agent can answer graph questions that exceed its prebuilt retrieval tools.

Key terms: benchmark dataset, end-to-end test examples, question, ground truth response, Cypher queries, ground truth, static string, graph database, benchmark, greetings

8.2 Evaluation

8.2.1 Context recall

This section defines context recall as a binary, sentence-level check of whether a generated answer is supported by retrieved context, indicating retrieval effectiveness.

• Context recall meaning: Explain that context recall measures whether retrieval successfully captures the relevant information needed to answer a query, with a high score indicating that necessary context was retrieved.
  Why it matters: It helps reveal whether weak RAG answers may result from missing relevant retrieved information.
• Sentence attribution evaluation: Describe the context recall evaluation prompt: for a given context and answer, it examines each answer sentence and classifies it as attributable to the context with Yes (1) or not attributable with No (0), returning JSON with reasoning.
  Why it matters: This provides a concrete way to assess whether the retrieved context supports the information appearing in an answer.
• Faithfulness distinction: Distinguish the stated role of context recall from faithfulness: context recall evaluates whether retrieval captures relevant information, while the following faithfulness assessment checks that the generated response remains factually aligned with retrieved content.
  Why it matters: Separating retrieval coverage from factual alignment helps diagnose whether a RAG problem lies in retrieval or generation.

Key terms: context recall, retrieval system, retrieved context, relevant information, Context recall evaluation, context, answer, binary classification, Yes (1), No (0)

8.2.2 Faithfulness

This section defines faithfulness as factual grounding in retrieved context and explains how to evaluate it by decomposing an answer into atomic statements and checking each statement against that context.

• Faithfulness metric: Explain faithfulness as a measure of whether every claim in a generated response is factually consistent with, and directly supported by, the retrieved context.
  Why it matters: Faithfulness evaluation helps detect hallucinated or unsupported claims in a RAG response.
• Atomic statement decomposition: Describe the first step of faithfulness assessment: breaking each answer sentence into one or more clear, self-contained atomic statements without pronouns, formatted as JSON.
  Why it matters: Splitting an answer into independently understandable claims makes each claim easier to verify against the context.
• Statement support verdicts: Trace the second step of faithfulness assessment: judge each atomic statement against the given context and return 1 when it can be directly inferred from the context or 0 when it cannot.
  Why it matters: Per-statement verdicts expose exactly which generated claims are grounded and which claims are unsupported.
• Faithfulness versus correctness: Distinguish faithfulness, which compares claims with retrieved context, from answer correctness, which compares the generated response with the ground truth.
  Why it matters: A response can be evaluated for grounding separately from whether it matches the expected answer.

Key terms: faithfulness, generated response, retrieved context, hallucination, atomic statements, Faithfulness statement breakdown, JSON, Faithfulness evaluation, context, verdict

8.2.3 Answer correctness

This section defines answer correctness as a ground-truth comparison that classifies answer statements to assess accuracy, completeness, and relevance.

• Answer correctness classification: Describe how answer correctness compares generated answer statements with ground-truth statements and classifies each statement as a true positive, false positive, or false negative.
  Why it matters: This classification shows whether an answer contains supported information and whether it omits expected information.
• Correctness evaluation purpose: Explain that answer correctness evaluates factual accuracy, completeness, and relevance to the user's intent, helping reveal missing context, inconsistencies, and inaccurate answers for iterative refinement.
  Why it matters: Evaluating these dimensions helps improve both retrieval of relevant data and the quality of generated responses.

Key terms: answer correctness, ground truth, answer statement, TP (true positive), FP (false positive), FN (false negative), factual accuracy, relevance, user's query, factual consistency

8.2.4 Loading the dataset

8.2.5 Running evaluation

This section runs the benchmark evaluation by generating answers and ground truth, recording retrieved context and latency, and scoring results with RAGAS.

• Benchmark evaluation inputs: Describe the inputs collected before evaluation: generated answers, expected ground truth responses, retrieved contexts, and latency. Explain that ground truth responses are obtained by executing the corresponding Cypher statements and generating answers with the agent.
  Why it matters: An evaluation can only assess the RAG system when its outputs are paired with expected answers and the retrieval and efficiency data needed for analysis.
• RAGAS scoring workflow: Explain that RAGAS runs the evaluation after the inputs are collected, requires non-null values, and replaces missing responses with “I don’t know.” State that it evaluates answer correctness, context recall, and faithfulness, after which the results are analyzed.
  Why it matters: The framework’s input requirement and metrics determine how the benchmark produces interpretable performance results.

Key terms: benchmark dataset, generated answers, ground truth responses, Cypher statements, retrieved contexts, latency, RAGAS framework, non-null values, I don’t know, answer correctness

8.2.6 Observations

This section interprets benchmark results to diagnose RAG weaknesses, identify targeted improvements, and support ongoing evaluation as the system evolves.

• Benchmark metric interpretation: Explain what the reported answer correctness, context recall, and faithfulness scores say about the agent’s performance. State that faithfulness is very high, while lower correctness and recall indicate that the system sometimes lacks needed retrieved information and consequently misses correct answers.
  Why it matters: Interpreting the metrics together helps you locate whether poor answers are caused by unsupported generation or incomplete retrieval.
• Retrieval-focused optimization: Describe why the benchmark results point to improving retrieval coverage as a route to better answer accuracy. Identify retrieval refinement, query reformulation, entity mapping for ambiguous queries, and improving how the LLM formulates answers as possible optimization directions.
  Why it matters: Metric-guided improvements let you address the component most likely responsible for weak answers instead of changing the system blindly.
• Response-level failure analysis: Describe how examining individual benchmark responses can reveal operational patterns and specific failure modes. Explain that queries avoiding text2cypher have lower latency because they avoid an additional LLM call, that LLM-as-judge scores can appear inconsistent, and that a failed longest-name question can be addressed with a few-shot text2cypher example or a dedicated tool.
  Why it matters: Individual examples expose problems that aggregate scores alone can hide, including latency costs, unreliable judgments, and unsupported query types.
• Growing benchmark dataset: Explain why the benchmark dataset should grow as the system evolves: it enables repeated measurement, reveals new weaknesses, and supports ongoing refinement of the knowledge graph RAG agent.
  Why it matters: A growing benchmark makes improvement measurable over time rather than relying on isolated examples or impressions.

Key terms: answer correctness, context recall, faithfulness, retrieved context, retrieval coverage, query reformulation, entity mapping, ambiguous queries, response accuracy, text2cypher

8.3 Next steps

This section positions the reader to build agentic GraphRAG systems by thoughtfully combining retrieval tools, embeddings, and knowledge-graph structure.

• Thoughtful retrieval tool design: Explain why increasingly capable LLMs still require thoughtfully designed, efficient, and well-integrated tools for retrieval, transformation, and manipulation. Relate tool design to the system's goals and constraints.
  Why it matters: Even a capable LLM can only use external retrieval and data tools as effectively as those tools are designed and integrated.
• Unstructured data retrieval options: Distinguish embedding unstructured text directly for fast similarity-based retrieval from extracting entities, relationships, and events to populate a knowledge graph for precise semantic and multihop queries. Describe how these approaches can be combined in one retrieval system.
  Why it matters: Choosing and combining these data approaches lets a GraphRAG system retrieve relevant information while supporting context-aware questions that require graph structure.
• Agentic GraphRAG readiness: Describe the practical next step presented in the section: applying the foundation from the book to build an agentic GraphRAG system, such as a question-answering agent or a domain-tailored retrieval pipeline.
  Why it matters: The prior techniques become useful when they are assembled into a retrieval system for a concrete application and domain.

Key terms: external tools, data retrieval, transformation, manipulation, goals and constraints, unstructured data, embed text, similarity-based retrieval, entities, relationships

8.4 Summary

This section summarizes how to benchmark a RAG pipeline with diverse tests and separate measures for retrieval, grounding, and answer quality.