A RAG system can answer thousands of questions against changing documents, making human review of every response or maintenance of reference answers expensive. Reference-free evaluation offers another route: an LLM judges the query, retrieved context, and generated response without a ground-truth answer. RAGAS applies this approach to separate retrieval problems from generation problems. section 14.2
The RAGAS evaluation triad asks three different questions:
- Faithfulness: Is the response grounded in the retrieved context? Claim extraction and verification help detect factually inconsistent answers. section 14.0 section 14.2.5
- Answer relevance: Does the response address the query? Reverse question generation helps expose answers that miss, underspecify, or incompletely address it. section 14.0
- Context relevance: Is the retrieved material focused on the query? Sentence extraction helps identify retrieval failures rather than blaming the generator. section 14.0
Evaluating a changing document collection
Suppose an enterprise assistant answers questions about evolving compliance documents. Evaluate each sampled query with its retrieved context and response; no curated answer is needed. A weak context-relevance score points toward retrieval, while weak faithfulness points toward the generated answer. Tracking the dimensions separately makes a quality complaint more actionable than a single overall score. section 14.2 section 14.0
Treat judge scores as evidence, not ground truth. Reported agreement with human annotators varies by dimension and evaluation setting; longer contexts challenge context-relevance judging, and a mostly grounded answer with one drifting clause is harder to catch than an entirely ungrounded answer. Judge model and prompt behavior can shift scores, while claim-level checks multiply LLM calls. Compare systems rather than overreading a small score change, and use an interval before acting on a difference. section 14.2.5 section 14.7



