A RAG assistant can retrieve the right documents and still produce an unsupported answer. Ragas evaluates the query, retrieved context, and response without requiring a reference answer. Its three metrics separate retrieval noise, unsupported generation, and answers that miss the question. section 14.2
Score the three relationships separately:
- Context relevance asks how much retrieved material is relevant to the query. A low score points toward noisy retrieval. section 14.2 section 14.2.4
- Faithfulness checks whether factual claims in the response are supported by the retrieved context. A low score flags unsupported claims even when the answer sounds plausible. section 14.2 section 14.5.1
- Answer relevance measures whether the response addresses the query at an appropriate level of specificity. It does not establish that the answer is grounded. section 14.2 section 14.2.4
A reapplication question
An advisor asks whether a client denied six months ago can reapply under new guidelines. Of 15 retrieved sentences, nine are relevant: context relevance is 0.60. Three of four answer claims are supported: faithfulness is 0.75, with an invented claim about automatic approval. Answer relevance is 0.87. The practical next steps are to investigate that claim and reduce retrieval noise; the on-topic answer alone is not a pass. section 14.2.4
Listing 14.2 The RAGAS triad, from the library.
import openai
from ragas.llms import llm_factory
from ragas.embeddings import OpenAIEmbeddings
from ragas.metrics.collections import (
Faithfulness, AnswerRelevancy, ContextRelevance)
client = openai.AsyncOpenAI() #A
llm = llm_factory("gpt-4o-mini", client=client)
emb = OpenAIEmbeddings(client=client,
model="text-embedding-3-small")
async def triad(question, answer, contexts):
"""Score one (question, answer, context) triple."""
f = await Faithfulness(llm=llm).ascore(
user_input=question, response=answer,
retrieved_contexts=contexts)
ar = await AnswerRelevancy(llm=llm, embeddings=emb).ascore(
user_input=question, response=answer)
cr = await ContextRelevance(llm=llm).ascore(
user_input=question, retrieved_contexts=contexts)
return {"faithfulness": f.value,
"answer_relevancy": ar.value,
"context_relevance": cr.value}
The Ragas 0.4.3 example scores one question, answer, and context set using asynchronous Faithfulness, AnswerRelevancy, and ContextRelevance calls. Pin the version before adapting the imports. section 14.2.4
Use scores to locate failures, not as unquestioned verdicts. In a two-answer comparison, answer relevance favored a fabricated answer while faithfulness exposed it. Judge scores also vary with model and prompt, require multiple LLM calls, and may miss subtle drift. Log the retrieved context, sample production queries, and watch trends rather than relying only on a fixed threshold. Ragas can also generate more challenging synthetic test questions, at additional cost. section 14.2.4 section 14.2.5 section 15.6.1 section 14.4.2



