A RAG assistant can retrieve the right policy and still give a generic answer to a specific question. Answer relevance measures whether the response addresses what the user asked. It penalizes off-topic, incomplete, overly vague, and unnecessarily padded answers. section 14.2.2
RAGAS scores answer relevance without a reference answer. An LLM generates several questions that the response could plausibly answer. The score averages the embedding similarity between those questions and the original query. A response at the wrong level of detail tends to produce questions at the wrong level of detail too. section 14.2.2
A reapplication question
Suppose an advisor asks whether a client with a 720 credit score, denied six months ago, can reapply under new guidelines. Three questions generated from the answer score 0.89, 0.82, and 0.91 against the original query, giving answer relevance of 0.87. The answer is on-topic, but that score alone does not establish that its claims are true: faithfulness separately tests support in the retrieved context. section 14.2.4
Listing 14.2 The RAGAS triad, from the library.
import openai
from ragas.llms import llm_factory
from ragas.embeddings import OpenAIEmbeddings
from ragas.metrics.collections import (
Faithfulness, AnswerRelevancy, ContextRelevance)
client = openai.AsyncOpenAI() #A
llm = llm_factory("gpt-4o-mini", client=client)
emb = OpenAIEmbeddings(client=client,
model="text-embedding-3-small")
async def triad(question, answer, contexts):
"""Score one (question, answer, context) triple."""
f = await Faithfulness(llm=llm).ascore(
user_input=question, response=answer,
retrieved_contexts=contexts)
ar = await AnswerRelevancy(llm=llm, embeddings=emb).ascore(
user_input=question, response=answer)
cr = await ContextRelevance(llm=llm).ascore(
user_input=question, retrieved_contexts=contexts)
return {"faithfulness": f.value,
"answer_relevancy": ar.value,
"context_relevance": cr.value}
The RAGAS example scores a question, answer, and retrieved contexts together. AnswerRelevancy takes the question and response; Faithfulness and ContextRelevance provide the separate checks needed to interpret its score. section 14.2.4
Use a low answer-relevance score to investigate information present in context but not extracted, incorrect specificity, or an incomplete response. Read it alongside context relevance and faithfulness before choosing a fix. A fabricated answer can score higher on answer relevance than a grounded one, so relevance is not a truth test. section 14.0 section 14.2.4
















