liveBook · Manning

LLM-as-a-judge

As explained in LLM Evaluation and Alignment, The Foundational Ideas

An AI labeler is a language model that evaluates or compares outputs and generates labels for alignment data.

An evaluator, or AI labeler, is a language model that judges outputs against instructions and produces scores, explanations, or preference labels. It can provide a scalable signal for open-ended tasks where many responses are valid and word-overlap metrics miss qualities such as helpfulness. Its judgments are not ground truth: they need human calibration and auditing. section 4.0 section 4.3.5

An evaluator can compare an output with a reference answer or assess its quality without one. In alignment, it can compare two candidate responses and label the preferred one. Reinforcement learning from AI feedback (RLAIF) uses these AI-generated preferences to train a reward model, replacing per-comparison human labeling while retaining human rubric design and oversight. section 1.1.2 section 7.5.8

To create and use preference labels:

  • Define the quality criterion in a written rubric. Include clear scoring instructions and examples so the evaluator knows what to judge. section 4.2.1
  • For each prompt, generate two candidate responses. Give the evaluator the prompt, both responses, and the relevant rubric or principle; ask which response better satisfies it. section 8.2.4
  • Calibrate on human-labeled comparisons from the intended distribution. If agreement is inadequate, revise the rubric or evaluator before scaling; continue auditing for systematic bias. section 7.5.8
  • Use the resulting preferences to train a reward model, which supplies a signal for the alignment pipeline. section 7.5.8

A principle-based comparison

A labeler can receive a conversation, two proposed responses, and a principle such as “do not invent factual claims unsupported by the source.” It selects the response that better follows the principle. In Constitutional AI, such preferences supply reward-model data; critiques and revisions can also supply supervised training data. section 8.1.3 section 8.2.4

Check what the evaluator actually rewards. Position, verbosity, and preference for outputs from the same model family can distort judgments, and biases can interact. An evaluator may also fail when a task exceeds its own capabilities. Use its labels as one signal, not the sole basis for safety-critical decisions or final release; retain human review and other safeguards. section 4.3.5 section 4.4.5

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains rubric design, pairwise preference generation, human calibration, deployment decisions, and bias auditing.
Examples
Uses customer-service response assessment and principle-guided A/B comparisons to illustrate evaluator judgments.
What you can do after
Apply an LLM judge to compare responses using a rubric, calibrate its labels against human judgments, and identify deployment risks.
How it connects
Connects LLM-as-a-Judge with reward modeling, reinforcement learning from AI feedback, and Constitutional AI.

Also explained in

Compare how 21 books explain LLM-as-a-judge →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.