liveBook · Manning

DeepEval

As explained in Retrieval Augmented Generation, The Foundational Ideas

DeepEval is a software framework for evaluating large language model applications and generating evaluation data.

DeepEval evaluates large language model applications, especially retrieval-augmented generation (RAG) systems, without requiring every judgment to be manually annotated. It supplies reusable metrics and synthetic evaluation data for testing reliability, coverage, and regression behavior. section 14.3.1 section 14.4.2

Its evaluation architecture combines LLM-based judging with structured criteria. A G-Eval metric defines a scoring rubric, prompts the judge through explicit evaluation steps, and returns a numeric result; DeepEval handles the surrounding prompt scaffolding and can use token probabilities when available. section 14.3.1

DeepEval supports several evaluation workflows:

  • Score faithfulness against source context using a production-ready G-Eval metric. section 14.3.1
  • Replace broad numeric bands with explicit rubric criteria when judge consistency requires sharper distinctions. section 14.3.1
  • Evaluate multi-turn transcripts with Conversation G-Eval when a single answer is not the appropriate unit. section 14.3.1
  • Generate document-grounded questions at configurable evolution depths, including cross-document reasoning tasks. section 14.4.2

A practical RAG evaluation loop

Provide documents to the Synthesizer, generate questions with varied complexity, and evaluate answers with G-Eval faithfulness scoring. Add unanswerable cases and successful production interactions to the test set so later changes become regression checks rather than relying only on aggregate scores. section 14.3.1 section 14.4.2

Listing 14.3 G-Eval-style faithfulness evaluator with chain-of-thought.
def geval_faithfulness(
    llm_client, context: str, answer: str
) -> dict[str, any]:
    """Evaluate faithfulness using G-Eval-style CoT prompting."""
    prompt = f"""You will assess the faithfulness of an answer
against its source context.
 
Evaluation Criteria:  #A
Faithfulness (1-5) - whether all claims in the answer
can be inferred from the context. A score of 5 means
every claim is fully supported. A score of 1 means
most claims are fabricated or contradicted.
 
Evaluation Steps:  #B
1. Identify each factual claim in the answer.
2. For each claim, check if the context supports it.
3. Count supported vs unsupported claims.
4. Assign a score from 1 to 5.  #C
 
Context: {context}
Answer: {answer}
 
Provide your evaluation steps, then your score (1-5)."""
    response = llm_client.generate(prompt)
    return {"evaluation": response}  #D
 

Shows the implementation pattern behind a G-Eval-style faithfulness evaluator: define criteria, expose evaluation steps, call an LLM client, and parse the returned score. section 14.3.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Provides implementation-ready metric and data-generation workflows, but leaves numeric-response parsing and production test-set design to the user.
Examples
Uses faithfulness scoring, configurable question evolution, cross-document questions, unanswerable cases, and captured successful interactions.
What it assumes
Assumes an LLM client and evaluation inputs such as context, answers, or documents, while supplying prompt scaffolding and generation workflows.
What you can do after
Apply it via G-Eval metrics and the Synthesizer to build broader RAG test suites and detect regressions.
How it connects
Connects G-Eval, RAG evaluation, chain-of-thought prompting, rubric scoring, synthetic data, regression testing, and multi-turn evaluation.

Also explained in

Compare how 2 books explain DeepEval →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.