Ragas

As explained in Building Reliable AI Systems

RAGAS is an open-source framework for evaluating retrieval-augmented generation systems.

from datasets import load_dataset
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from ragas import evaluate #1
from ragas.metrics import (
    answer_relevancy,
    faithfulness,
    context_recall,
    context_precision,
)

amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")
amnesty_qa

result = evaluate(
    amnesty_qa["eval"],
    metrics=[
        context_precision,
        faithfulness,
        answer_relevancy,
        context_recall,
    ],
    llm=ChatOpenAI(model="gpt-4o-mini"),
    embeddings=OpenAIEmbeddings(),
    column_map={"user_input": "question", "response": "answer",
               "retrieved_contexts": "contexts", "reference": "ground_truth"},
)

print(results)

How Building Reliable AI Systems covers it

How deep it goes

Demonstrates dataset loading and metric configuration, but does not explain metric internals.

Examples

Uses the Amnesty QA dataset with a column map, GPT-4o-mini, and OpenAI embeddings.

What you can do after

Apply Ragas to evaluate a dataset using selected metrics, an LLM, embeddings, and mapped fields.

How it connects

Connects RAG evaluation with context precision, faithfulness, answer relevancy, and context recall.

liveBookmanning.com

Read Building Reliable AI Systems on liveBook Get the book at manning.com

Related concepts

• fidelityuses · Fidelity is the degree to which an output, model, or representation accurately reflects or is supported by its relevant source, target, evidence, or behavior.
• retrieval-augmented generationmeasures · Retrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.
• relevancehas part · Relevance is the degree to which information, a result, or a response pertains to a query, task, purpose, or information need.
• language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• synthetic dataproduces · Synthetic data is data generated artificially rather than collected directly from the target process, often to approximate its properties or meet specified requirements.

This explainer was generated by Manning's knowledge graph from the text of Building Reliable AI Systems. The section links point to the parts of the book it draws on.