liveBook · Manning

Ragas

As explained in Retrieval Augmented Generation, The Foundational Ideas

RAGAS is a framework for evaluating retrieval-augmented generation systems with metrics including faithfulness, answer relevance, and context relevance.

A RAG assistant can retrieve the right documents and still produce an unsupported answer. Ragas evaluates the query, retrieved context, and response without requiring a reference answer. Its three metrics separate retrieval noise, unsupported generation, and answers that miss the question. section 14.2

Score the three relationships separately:

  • Context relevance asks how much retrieved material is relevant to the query. A low score points toward noisy retrieval. section 14.2 section 14.2.4
  • Faithfulness checks whether factual claims in the response are supported by the retrieved context. A low score flags unsupported claims even when the answer sounds plausible. section 14.2 section 14.5.1
  • Answer relevance measures whether the response addresses the query at an appropriate level of specificity. It does not establish that the answer is grounded. section 14.2 section 14.2.4

A reapplication question

An advisor asks whether a client denied six months ago can reapply under new guidelines. Of 15 retrieved sentences, nine are relevant: context relevance is 0.60. Three of four answer claims are supported: faithfulness is 0.75, with an invented claim about automatic approval. Answer relevance is 0.87. The practical next steps are to investigate that claim and reduce retrieval noise; the on-topic answer alone is not a pass. section 14.2.4

Listing 14.2 The RAGAS triad, from the library.
import openai
from ragas.llms import llm_factory
from ragas.embeddings import OpenAIEmbeddings
from ragas.metrics.collections import (
    Faithfulness, AnswerRelevancy, ContextRelevance)
client = openai.AsyncOpenAI()  #A
llm = llm_factory("gpt-4o-mini", client=client)
emb = OpenAIEmbeddings(client=client,
                       model="text-embedding-3-small")
async def triad(question, answer, contexts):
    """Score one (question, answer, context) triple."""
    f = await Faithfulness(llm=llm).ascore(
        user_input=question, response=answer,
        retrieved_contexts=contexts)
    ar = await AnswerRelevancy(llm=llm, embeddings=emb).ascore(
        user_input=question, response=answer)
    cr = await ContextRelevance(llm=llm).ascore(
        user_input=question, retrieved_contexts=contexts)
    return {"faithfulness": f.value,
            "answer_relevancy": ar.value,
            "context_relevance": cr.value}
 

The Ragas 0.4.3 example scores one question, answer, and context set using asynchronous Faithfulness, AnswerRelevancy, and ContextRelevance calls. Pin the version before adapting the imports. section 14.2.4

Use scores to locate failures, not as unquestioned verdicts. In a two-answer comparison, answer relevance favored a fabricated answer while faithfulness exposed it. Judge scores also vary with model and prompt, require multiple LLM calls, and may miss subtle drift. Log the retrieved context, sample production queries, and watch trends rather than relying only on a fixed threshold. Ragas can also generate more challenging synthetic test questions, at additional cost. section 14.2.4 section 14.2.5 section 15.6.1 section 14.4.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Works through all three scores and shows library calls, then qualifies their reliability rather than treating scores as definitive.
Examples
Uses a financial-services reapplication query and a grounded-versus-fabricated answer comparison to expose disagreement between metrics.
What it assumes
Requires a query, retrieved context, and generated response, but not curated ground-truth answers.
What you can do after
Apply it via Ragas to score query–context–response triples and investigate which pipeline stage needs attention.
How it connects
Connects LLM-as-judge evaluation to retrieval, hallucination detection, synthetic test generation, and production monitoring.

Also explained in

Compare how 4 books explain Ragas →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.