liveBook · Manning

LLM-as-a-judge

As explained in Retrieval Augmented Generation, The Foundational Ideas

An LLM judge is a large language model configured to evaluate the quality or correctness of generated outputs.

A RAG system can retrieve the right documents and still produce an answer that drifts away from them. An LLM judge uses a language model to assess outputs or retrieved context, making failures visible that retrieval scores alone can miss. Its verdict is a measurement, not ground truth: a passing threshold can hide a worsening trend. section 14.0

The judge’s job depends on where it sits in the pipeline:

  • In evaluation, prompted judges score RAG behavior; RAGAS can do so without reference answers, while G-Eval adds chain-of-thought reasoning to fine-grained scoring. section 14.3 section 14.3.3
  • In Self-RAG training, a stronger critic labels examples with reflection tokens. A smaller student learns from those labeled examples and later emits its own control signals; the critic is not required for each student prediction. section 9.3 section 9.3.1
  • In dual-path graph retrieval, a critic checks merged graph and vector context against the question. It can reject inadequate context and send the query back for another routing pass before answer generation. section 12.4.2

A retrieval improvement that masked an answer regression

A mortgage lender’s compliance assistant gained a cross-encoder reranker and retrieved better documents, yet answer faithfulness fell. The existing threshold did not flag the decline; comparing reports across weeks exposed it. Judge-based scoring is most useful here when its results are examined as diagnostic evidence rather than accepted because a score passes. section 14.0

Judge choice carries tradeoffs. A frontier critic is an option when API cost and data sensitivity permit; open-weight or domain-specific critics can suit privacy or specialized notions of support. Prompted evaluation needs no training, whereas the compared ARES approach trains a small evaluator using synthetic data and annotations. Neither sophistication nor fluency removes bias: judges can favor familiar or self-generated wording, formatting, or a position in a pair, and repeated runs can disagree. Check the judge’s behavior on the task at hand. section 9.3.1 section 14.0 section 14.3.3

Table 14.2 Comparing RAG evaluation approaches
Approach
Judge type
Training required
Confidence bounds
Cost per sample
Best for
RAGAS
Prompted LLM
None
No
High (multiple LLM calls)
Quick evaluation, prototyping
G-Eval
Prompted LLM + CoT
None
No
Medium (one LLM call + CoT)
Fine-grained scoring with CoT reasoning
ARES
Fine-tuned DeBERTa
Synthetic data + 150 annotations
Yes (PPI)
Low (small model inference)
Production monitoring, rigorous comparison

Compare prompted RAGAS and G-Eval with trained ARES by training needs, confidence bounds, sample cost, and intended use. section 14.3.3

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Distinguishes prompted, trained, and pipeline critics, then examines selection tradeoffs and documented biases without specifying a universal grading rubric.
Examples
Uses a mortgage-compliance faithfulness regression, Self-RAG’s critic-labeled training data, and a graph-retrieval retry path.
What it assumes
Assumes familiarity with generated answers and retrieval, while defining behavioral distillation as learning from a stronger model’s labeled examples.
What you can do after
Evaluate/understand when a judge can diagnose RAG failures or control a pipeline, and check its bias before trusting scores.
How it connects
Links LLM judging to RAG evaluation, faithfulness, behavioral distillation, reflection tokens, graph retrieval, and vector search.

Also explained in

Compare how 21 books explain LLM-as-a-judge →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.