liveBook · Manning

LLM-as-a-judge

As explained in Build an Advanced RAG Application (From Scratch)

LLM-as-judge evaluation uses a language model to assess the quality, factual consistency, or other properties of generated responses.

LLM-as-judge evaluation uses a language model to assess generated responses for quality, factual consistency, and hallucination. In RAGAS, faithfulness is computed with this approach, producing more nuanced judgments than rule-based systems. section 6.5.2

The pattern supports RAG evaluation by:

  • Assessing whether generated answers remain factually grounded in retrieved information. section 6.5.2
  • Providing nuanced quality judgments beyond basic or rule-based metrics. section 6.5.2
  • Supporting systematic comparison of retrieval and generation changes during production-aligned testing. section 6.5.2

RAGAS faithfulness

RAGAS uses an LLM-as-judge approach to compute faithfulness, while HHEM uses a classification model for the same hallucination and factual-consistency concerns. section 6.5.2

Figure 6.7 RAGAS Core Metric extensively covers LLM Grounding and Response (Source)

Use the figure to inspect how RAGAS metrics connect grounding with generated-response evaluation. section 6.5.2

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Contrasts model-based judgment with classification and rule-based evaluation but does not specify prompts or implementation steps.
Examples
Uses RAGAS faithfulness, HHEM comparison, Vertex AI evaluation, and production-aligned metric testing as concrete evaluation contexts.
What you can do after
Evaluate RAG responses with nuanced model-based judgments and compare system changes against selected metrics.
How it connects
Connects RAG evaluation with faithfulness, hallucination detection, factual consistency, retrieval, and generation.

Also explained in

Compare how 21 books explain LLM-as-a-judge →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.