LLM-as-a-judge uses a separate, typically more capable language model to assess a candidate response against a rubric. It is useful for semantic qualities such as faithfulness and relevance, which simple string-matching heuristics can miss. section 8.2.5 section 8.2.2
The evaluation prompt supplies the user query, candidate output, and any context or reference material needed for scoring. The judge acts as an impartial evaluator and returns a numerical score with a short justification. Structured output, such as JSON, lets an automated pipeline consume the result across many test cases. section 8.2.5
Choose a judging pattern to match the decision:
- Direct scoring rates one output against a rubric, for criteria such as factual accuracy, instruction following, format compliance, or groundedness. section 8.2.5
- Pairwise comparison selects between two outputs, for preferences such as tone, clarity, or persuasiveness. Run the comparison in both orders and check that the preference remains stable. section 8.2.5
- A decision-tree rubric asks closed questions in sequence; the path through the tree provides an auditable explanation of the score. section 8.2.2
- Question-answer generation turns individual claims into closed-ended questions, answers them against reference context, and scores the proportion verified. section 8.2.2
Evaluating a RAG pipeline
For a small benchmark dataset, use Ragas to evaluate faithfulness and answer relevancy alongside context precision and context recall. Supply the evaluation dataset, metrics, LLM, and embeddings to evaluate; the resulting scores give signals about generation and retrieval quality. Ragas APIs vary across releases, so pin the version used by the implementation. section 8.4.3 section 8.2.2
Listing 8.4 Python example for evaluation using the LLM-as-judge pattern with Ragas
from ragas import evaluate
from ragas.metrics import answer_relevancy, context_precision,
context_recall, faithfulness
# Build RAG pipeline and an evaluation dataset (details in the repo)
metrics = [ #1
faithfulness,
answer_relevancy,
context_recall,
context_precision,
]
result = evaluate( #2
dataset=eval_dataset,
metrics=metrics,
llm=llm,
embeddings=embeddings,
raise_exceptions=False,
)
The listing shows the Ragas metrics and evaluation call used in the example. section 8.4.3
Judge scores are decision-support signals, not objective truth. Position bias can favor one candidate’s placement, verbosity bias can reward unnecessary length, and self-enhancement bias can favor outputs from the judge’s own model family. Keep human oversight and calibrate scores against deterministic checks or human-labeled examples before using them as release gates. section 8.2.5 section 8.4.3




















