DeepEval evaluates large language model applications, especially retrieval-augmented generation (RAG) systems, without requiring every judgment to be manually annotated. It supplies reusable metrics and synthetic evaluation data for testing reliability, coverage, and regression behavior. section 14.3.1 section 14.4.2
Its evaluation architecture combines LLM-based judging with structured criteria. A G-Eval metric defines a scoring rubric, prompts the judge through explicit evaluation steps, and returns a numeric result; DeepEval handles the surrounding prompt scaffolding and can use token probabilities when available. section 14.3.1
DeepEval supports several evaluation workflows:
- Score faithfulness against source context using a production-ready G-Eval metric. section 14.3.1
- Replace broad numeric bands with explicit rubric criteria when judge consistency requires sharper distinctions. section 14.3.1
- Evaluate multi-turn transcripts with Conversation G-Eval when a single answer is not the appropriate unit. section 14.3.1
- Generate document-grounded questions at configurable evolution depths, including cross-document reasoning tasks. section 14.4.2
A practical RAG evaluation loop
Provide documents to the Synthesizer, generate questions with varied complexity, and evaluate answers with G-Eval faithfulness scoring. Add unanswerable cases and successful production interactions to the test set so later changes become regression checks rather than relying only on aggregate scores. section 14.3.1 section 14.4.2
Listing 14.3 G-Eval-style faithfulness evaluator with chain-of-thought.
def geval_faithfulness(
llm_client, context: str, answer: str
) -> dict[str, any]:
"""Evaluate faithfulness using G-Eval-style CoT prompting."""
prompt = f"""You will assess the faithfulness of an answer
against its source context.
Evaluation Criteria: #A
Faithfulness (1-5) - whether all claims in the answer
can be inferred from the context. A score of 5 means
every claim is fully supported. A score of 1 means
most claims are fabricated or contradicted.
Evaluation Steps: #B
1. Identify each factual claim in the answer.
2. For each claim, check if the context supports it.
3. Count supported vs unsupported claims.
4. Assign a score from 1 to 5. #C
Context: {context}
Answer: {answer}
Provide your evaluation steps, then your score (1-5)."""
response = llm_client.generate(prompt)
return {"evaluation": response} #D
Shows the implementation pattern behind a G-Eval-style faithfulness evaluator: define criteria, expose evaluation steps, call an LLM client, and parse the returned score. section 14.3.1

