liveBook · Manning

Relevance

As explained in Retrieval Augmented Generation, The Foundational Ideas

Answer relevance is a metric measuring how well a generated answer addresses the user's question.

A RAG assistant can retrieve the right policy and still give a generic answer to a specific question. Answer relevance measures whether the response addresses what the user asked. It penalizes off-topic, incomplete, overly vague, and unnecessarily padded answers. section 14.2.2

RAGAS scores answer relevance without a reference answer. An LLM generates several questions that the response could plausibly answer. The score averages the embedding similarity between those questions and the original query. A response at the wrong level of detail tends to produce questions at the wrong level of detail too. section 14.2.2

A reapplication question

Suppose an advisor asks whether a client with a 720 credit score, denied six months ago, can reapply under new guidelines. Three questions generated from the answer score 0.89, 0.82, and 0.91 against the original query, giving answer relevance of 0.87. The answer is on-topic, but that score alone does not establish that its claims are true: faithfulness separately tests support in the retrieved context. section 14.2.4

Listing 14.2 The RAGAS triad, from the library.
import openai
from ragas.llms import llm_factory
from ragas.embeddings import OpenAIEmbeddings
from ragas.metrics.collections import (
    Faithfulness, AnswerRelevancy, ContextRelevance)
client = openai.AsyncOpenAI()  #A
llm = llm_factory("gpt-4o-mini", client=client)
emb = OpenAIEmbeddings(client=client,
                       model="text-embedding-3-small")
async def triad(question, answer, contexts):
    """Score one (question, answer, context) triple."""
    f = await Faithfulness(llm=llm).ascore(
        user_input=question, response=answer,
        retrieved_contexts=contexts)
    ar = await AnswerRelevancy(llm=llm, embeddings=emb).ascore(
        user_input=question, response=answer)
    cr = await ContextRelevance(llm=llm).ascore(
        user_input=question, retrieved_contexts=contexts)
    return {"faithfulness": f.value,
            "answer_relevancy": ar.value,
            "context_relevance": cr.value}
 

The RAGAS example scores a question, answer, and retrieved contexts together. AnswerRelevancy takes the question and response; Faithfulness and ContextRelevance provide the separate checks needed to interpret its score. section 14.2.4

Use a low answer-relevance score to investigate information present in context but not extracted, incorrect specificity, or an incomplete response. Read it alongside context relevance and faithfulness before choosing a fix. A fabricated answer can score higher on answer relevance than a grounded one, so relevance is not a truth test. section 14.0 section 14.2.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains reverse-question generation, embedding similarity, and averaged scores, then works a numerical example without deriving embedding internals.
Examples
Works a mortgage reapplication query with similarities of 0.89, 0.82, and 0.91, yielding an answer-relevance score of 0.87.
What it assumes
Requires an original query and generated response, but no curated reference answer or human-annotated ground truth.
What you can do after
Apply it via RAGAS to flag off-topic or wrongly specific answers, while checking faithfulness before trusting a relevant response.
How it connects
Links answer relevance to RAGAS, faithfulness, context relevance, embedding similarity, and retrieval-augmented generation failure diagnosis.

Also explained in

Compare how 17 books explain relevance →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.