liveBook · Manning

LLM-as-a-judge

As explained in Context Engineering

An evaluator is a component that assesses outcomes against specified criteria or metrics.

LLM-as-a-judge uses a separate, typically more capable language model to assess a candidate response against a rubric. It is useful for semantic qualities such as faithfulness and relevance, which simple string-matching heuristics can miss. section 8.2.5 section 8.2.2

The evaluation prompt supplies the user query, candidate output, and any context or reference material needed for scoring. The judge acts as an impartial evaluator and returns a numerical score with a short justification. Structured output, such as JSON, lets an automated pipeline consume the result across many test cases. section 8.2.5

Choose a judging pattern to match the decision:

  • Direct scoring rates one output against a rubric, for criteria such as factual accuracy, instruction following, format compliance, or groundedness. section 8.2.5
  • Pairwise comparison selects between two outputs, for preferences such as tone, clarity, or persuasiveness. Run the comparison in both orders and check that the preference remains stable. section 8.2.5
  • A decision-tree rubric asks closed questions in sequence; the path through the tree provides an auditable explanation of the score. section 8.2.2
  • Question-answer generation turns individual claims into closed-ended questions, answers them against reference context, and scores the proportion verified. section 8.2.2

Evaluating a RAG pipeline

For a small benchmark dataset, use Ragas to evaluate faithfulness and answer relevancy alongside context precision and context recall. Supply the evaluation dataset, metrics, LLM, and embeddings to evaluate; the resulting scores give signals about generation and retrieval quality. Ragas APIs vary across releases, so pin the version used by the implementation. section 8.4.3 section 8.2.2

Listing 8.4 Python example for evaluation using the LLM-as-judge pattern with Ragas
from ragas import evaluate
from ragas.metrics import answer_relevancy, context_precision,
context_recall, faithfulness

# Build RAG pipeline and an evaluation dataset (details in the repo)

metrics = [ #1
    faithfulness,
    answer_relevancy,
    context_recall,
    context_precision,
]

result = evaluate( #2
    dataset=eval_dataset,
    metrics=metrics,
    llm=llm,
    embeddings=embeddings,
    raise_exceptions=False,
)

The listing shows the Ragas metrics and evaluation call used in the example. section 8.4.3

Judge scores are decision-support signals, not objective truth. Position bias can favor one candidate’s placement, verbosity bias can reward unnecessary length, and self-enhancement bias can favor outputs from the judge’s own model family. Keep human oversight and calibrate scores against deterministic checks or human-labeled examples before using them as release gates. section 8.2.5 section 8.4.3

How Context Engineering covers it

How deep it goes
Explains judge inputs, structured outputs, scoring patterns, biases, and a library-based evaluation workflow.
Examples
Evaluates a small RAG benchmark using faithfulness, answer relevancy, context precision, and context recall.
What you can do after
Apply it via Ragas to generate evaluation signals and calibrate them before using them as release gates.
How it connects
Connects semantic scoring with retrieval-augmented generation, faithfulness, relevance, and pairwise comparison.

Also explained in

Compare how 21 books explain LLM-as-a-judge →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Context Engineering. The section links point to the parts of the book it draws on.