liveBook · Manning

Evaluator

As explained in Reinforcement Learning from Human Feedback

LLM-as-a-judge is an evaluation technique in which a language model assesses the quality or preference of model outputs according to provided criteria.

LLM-as-a-judge evaluates model outputs by prompting a language model with judging instructions, an input prompt, and competing completions. It replaces or augments human preference labeling for scalable evaluation and synthetic preference data, reducing collection costs while introducing evaluator bias and variance. section 5.7 section 12.4

[System]
Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below.
You should choose the assistant that follows the user's instructions and answers the user's question better.
Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses.
Begin your evaluation by comparing the two responses and provide a short explanation.
Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision.
Do not allow the length of the responses to influence your evaluation.
Do not favor certain names of the assistants.
Be as objective as possible.
After providing your explanation, output your final verdict by strictly following this format: "[[A]]" if assistant A is better, "[[B]]" if assistant B is better, and "[[C]]" for a tie.
[User Question]
{question}
[The Start of Assistant A's Answer]
{answer_a}
[The End of Assistant A's Answer]
[The Start of Assistant B's Answer]
{answer_b}
[The End of Assistant B's Answer]

Use an impartial pairwise-judging prompt that specifies evaluation criteria, suppresses position and length bias, and requires a constrained final verdict. section 5.7

A practical judging workflow typically includes:

  • Provide the user request, two candidate responses, and explicit criteria such as helpfulness, relevance, accuracy, depth, creativity, and detail. section 5.7
  • Request a comparison followed by a machine-readable preference label, enabling preference-pair construction or benchmark scoring. section 5.7 section 5.9
  • Use temperature 0 to reduce rating variance, then inspect for inconsistency, self-preference, position bias, and length bias. section 5.7 section 12.4.2

Chat benchmark comparison

MT-Bench, AlpacaEval, and Arena-Hard use LLM judges to compare chat responses at scale. Their scores can be distorted when longer answers receive higher ratings regardless of quality, motivating length-controlled variants and cautious interpretation. section 16.0 section 16.6

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Explains pairwise prompts, temperature control, and major biases, but stops short of a settled calibration recipe.
Examples
Uses judge prompts, MT-Bench, AlpacaEval, Arena-Hard, pairwise completions, and length-controlled evaluation variants.
What it assumes
Assumes familiarity with model outputs, preference comparisons, reward modeling, and post-training pipelines.
What you can do after
Apply it via structured prompts to compare outputs, generate preference pairs, and scale evaluations while checking bias and uncertainty.
How it connects
Connects preference data, reward modeling, RLHF, RLAIF, synthetic data, and evaluation benchmarks.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Also explained in

Compare how 9 books explain evaluator →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.