Chapter 5: Acquiring judgments
This introduction defines judgments, explains their role in evaluation metrics, and shows why judgment quality affects evaluation results.
After this chapter you will be able to
- Judgments and their forms. Describe what a judgment is in AI evaluation and give examples of forms it can take.
- Judgments in evaluation metrics. Distinguish how judgment-based and reference-based metrics rely on judgments.
- Judgment quality and distortion. Explain how judgment quality can affect evaluation results and give an example of a source of distortion.
- Binary labels and nuance. Explain the trade-off between simple binary judgments and capturing nuances of correctness and safety in medical advice.
- Human judgment scalability trade-offs. Describe the scalability concern with relying on doctors and the proposed use of trained non-experts, including concerns about ambiguous rubrics and disagreement.
- LLM judge defensibility. Explain why the speed and cost advantages of an LLM judge do not by themselves make its judgments defensible.
- Binary and scalar judgments. Distinguish binary judgments from scalar judgments using an evaluation example.
- Absolute and relative judgments. Distinguish judgments of an output against a standard from direct comparisons among outputs.
- Independent format dimensions. Explain how the two judgment-format choices combine.
- Binary and scalar suitability. Distinguish when binary judgments are appropriate from when scalar judgments better capture the evaluation criterion.
- Structured binary checks. Explain how several concrete binary checks can replace a subjective overall scalar rating, including the preparation required and the potential benefits.
- Advantages of relative judgments. Explain why comparing outputs can be easier and more sensitive than assigning absolute scores, and when that sensitivity may break down.
- Comparison versus absolute standards. Distinguish situations suited to relative judgments from those requiring absolute judgments.
- Rubric purpose and scope. Describe what a rubric specifies and where it is used to acquire judgments.
- Rubric quality and consistency. Explain how rubric quality affects the consistency and focus of evaluation judgments.
- Observable construct criteria. Explain how to break an abstract evaluation construct into properties or behaviors that judges can inspect, using helpfulness as an example.
- Necessary and optional components. Distinguish criteria essential to a construct from optional enhancements, and explain why the distinction matters.
- Failure mode checks. Describe how to turn common task failures into explicit checks that judges can verify.
- Independent non-overlapping criteria. Explain why criteria should be independent and non-overlapping, using clarity and coherence as an example.
- Positive and negative anchors. Explain how prototypical positive examples and counterexamples clarify an evaluation criterion.
- Borderline judgment thresholds. Explain how borderline examples help judges apply a rubric consistently.
- Contrastive dimension examples. Explain how contrastive examples isolate the dimension being judged.
- Representative rubric examples. Explain why rubric examples should reflect the data judges will encounter.
- Operational task instructions. Explain how to define a judgment task in concrete terms, including what to evaluate, what to ignore, and the expected answer format.
- Permitted information boundaries. Explain why instructions should state what information judges may and may not use, using the medical chatbot example.
- Misunderstanding and disagreement clarifications. Explain how to identify recurring judge misunderstandings and address them explicitly in instructions.
- Judge selection and evaluation validity. Explain why the choice of judge should match the judgment task and what can happen when it does not.
- Expert judgment strengths and limitations. Explain what domain experts contribute to evaluation and why relying on them can be difficult at scale.
- Expert investment and hybrid roles. Justify when to invest in domain experts and describe roles they can play alongside other judges.
- Sources of expert disagreement. Explain why experts can disagree and what that disagreement can reveal.
- Trained non-expert capabilities. Explain how trained non-experts can provide reliable judgments and what advantages they offer over domain experts.
- Training and quality limitations. Describe the preparation and monitoring non-expert judges need and the cases they may struggle to judge.
- Suitable tasks and workflows. Explain when non-expert annotators are a good fit and how they can be used alongside experts or to pilot judgment pipelines.
- Crowdsourcing benefits and suitability. Explain how crowdsourced judging works, its advantages, and the kinds of tasks it suits.
- Crowdsourcing quality and skew. Explain why crowdsourced judgments can be unreliable or unrepresentative and why quality control is necessary.
- Sensitive data protection. Describe the data-protection responsibilities involved when human judges encounter sensitive information.
- AI judge role. Explain what AI-as-a-judge does and how it can be instructed to provide judgments.
- Scalability and evaluation uses. Explain why AI judges are useful for large-volume evaluations and give an example of an enabled use.
- Sensitivity and judgment biases. Describe how changes in evaluation setup and model biases can distort AI judgments.
- Human validation and oversight. Explain what evidence supports AI judging and why human validation and review remain necessary.
- Individual judge practice. Describe how practice rounds help judges learn the task and rubric before full-scale judging, and give examples of activities that support learning and clarification.
- Group calibration consistency. Explain the purpose of calibration rounds and distinguish it from the purpose of practice rounds.
- Baseline LLM judge prompt. Describe how to start building an LLM judge and how the baseline relates to the evaluation’s reliability needs.
- Judgment format and reliability. Explain how judgment format can affect LLM judge reliability and how to choose a starting format.
- Generalist and specialist judges. Explain when a general-purpose LLM is sufficient and when a specialized judge may be worth considering.
- Judge strength trade-offs. Explain the benefits of stronger judge models and the practical reasons a weaker model may be appropriate.
- Starting strength strategies. Compare starting with a weaker judge and starting with a stronger judge when selecting model strength.
- Explicit judgment instructions. Explain how to translate a rubric into explicit instructions for an LLM judge, including relevant rules and, where appropriate, a structured reasoning process.
- Rigid and separated outputs. Describe how to constrain the judge's output format and separate the final decision from any explanation.
- Permitted information sources. Explain why a judgment prompt should identify which information sources the model may and may not use.
- Bias and shortcut safeguards. Explain how to prompt the judge to evaluate examples independently and avoid superficial cues, guesswork, and partial evaluation.
- Human Reference Validation. Describe how to set up a validation comparison between an LLM judge and human evaluators, and explain why humans serve as the reference.
- Judgment Format Agreement Measures. Describe how agreement measures differ for binary, scalar, and pairwise judgments.
- Disagreement Driven Refinement. Trace the iterative process for improving an LLM judge when its agreement with humans is unsatisfactory.
- Human Judgment Validation Design. Explain how the experiment used human-labeled SNLI examples to validate the two LLM judges, including how it controlled the comparison.
- Toolkit Faithfulness Definitions. Distinguish the two toolkits' definitions of faithfulness and explain how they affect the expected judgment for a neutral example.
- Relationship Type Results. Describe how the judges' results differed on entailment, neutral, and contradiction cases, and what those differences suggest.
- Judgment workflow scope. Describe what a judgment workflow covers, from assigning items to judges through handling assessments and maintaining quality.
- Single and multiple judges. Explain the trade-offs between assigning one judge and multiple judges to each item, including why independent errors matter.
- Pilot-based judge count. Describe how to use a pilot batch and a higher-confidence reference to choose the number of judges.
- Stratified judge allocation. Explain how stratified allocation assigns judges to predefined groups and what it depends on.
- Adaptive judge allocation. Describe how adaptive allocation escalates cases using available signals and explain its routing risk.
- Fully parallel judgment workflows. Explain how fully parallel judging works and its trade-off between independence and efficiency.
- Fully sequential judgment workflows. Explain how sequential judging works and why it can save effort while exposing later judges to influence.
- Hybrid review selection strategies. Describe how a hybrid workflow combines independent judgments with selective review, and distinguish disagreement-based escalation from sampling.
- Judgment aggregation strategies. Explain how judgment format and trust in judges affect the choice of aggregation strategy.
- Disagreement signals and thresholds. Explain why disagreement can be informative and how tolerance rules distinguish routine variation from disagreement that warrants attention.
- Responses to elevated disagreement. Distinguish adjudication for an individual disputed item from structured analysis of recurring disagreement across items.
- Ongoing judgment quality monitoring. Explain why judgment quality needs active monitoring over time, including how problems can develop without an obvious individual error.
What it builds on
- What is a metric (chapter 3) · The basic idea of an evaluation metric.
- Choosing evaluation dimensions (chapter 2) · Evaluation dimensions identify qualities such as correctness and safety to judge.
- Designing custom metrics (chapter 3) · Designing custom evaluation metrics
- Correlation with human judgments (chapter 3) · Correlation with human judgments is a consideration when evaluating metrics.
Section by section
Introduction
This introduction defines judgments, explains their role in evaluation metrics, and shows why judgment quality affects evaluation results.
- Judgments and their forms: Describe what a judgment is in AI evaluation and give examples of forms it can take.
Why it matters: Recognizing the different forms of judgments helps you identify what an evaluation asks judges to assess. - Judgments in evaluation metrics: Distinguish how judgment-based and reference-based metrics rely on judgments.
Why it matters: Judgments can affect an evaluation even when the metric uses pre-existing reference data. - Judgment quality and distortion: Explain how judgment quality can affect evaluation results and give an example of a source of distortion.
Why it matters: Flawed judgments can lead you to draw the wrong conclusion about a system.
Key terms: judgment, evaluation criterion, label, rating, ranking, preference, judgment-based metrics, reference-based metrics, reference answers, gold labels
5.1 “Who and how decides what’s correct?”
The section introduces judgment-acquisition trade-offs for evaluating a medical advice chatbot.
- Binary labels and nuance: Explain the trade-off between simple binary judgments and capturing nuances of correctness and safety in medical advice.
Why it matters: The judgment format affects whether important safety and correctness distinctions remain visible. - Human judgment scalability trade-offs: Describe the scalability concern with relying on doctors and the proposed use of trained non-experts, including concerns about ambiguous rubrics and disagreement.
Why it matters: Choosing human judges means balancing repeatability and scale against consistent interpretation and disagreement. - LLM judge defensibility: Explain why the speed and cost advantages of an LLM judge do not by themselves make its judgments defensible.
Why it matters: Fast, consistent judgments still need to be defensible when they affect consequential decisions.
Key terms: judgments, binary labels, correct, safe, domain experts, trained non-experts, rubric, redundancy, disagreement, LLM as a judge
5.2 Deciding what judgments should look like
This section introduces how evaluation judgments are shaped by independent choices between binary or scalar and absolute or relative formats.
- Judgment format selection: Explain when judgment format is chosen and how selected metrics influence that choice.
Why it matters: The format of judgments needs to fit what the evaluation intends to measure. - Binary and scalar judgments: Distinguish binary judgments from scalar judgments using an evaluation example.
Why it matters: This choice determines whether judgments record a simple classification or a more nuanced rating. - Absolute and relative judgments: Distinguish judgments of an output against a standard from direct comparisons among outputs.
Why it matters: This distinction determines whether an output is judged on its own or against alternatives. - Independent format dimensions: Explain how the two judgment-format choices combine.
Why it matters: Choosing one dimension does not settle how the judgment works on the other.
Key terms: judgment format, metrics, custom metrics, binary judgments, scalar judgments, numerical scale, ordinal scale, absolute judgments, relative judgments, pairwise comparisons
5.2.1 Binary vs. scalar judgments
This section compares binary and scalar judgments and presents structured binary checks as an alternative to subjective overall ratings.
- Binary and scalar suitability: Distinguish when binary judgments are appropriate from when scalar judgments better capture the evaluation criterion.
Why it matters: Matching the judgment format to the criterion helps preserve the differences the evaluation needs to measure. - Scalar judgment practical limitations: Explain the practical difficulties and limitations of scalar judgments, including differences in how judges use scale points.
Why it matters: More rating levels do not automatically make judgments easier or more useful. - Structured binary checks: Explain how several concrete binary checks can replace a subjective overall scalar rating, including the preparation required and the potential benefits.
Why it matters: Concrete checks can make a nuanced judgment easier to perform and easier to audit.
Key terms: binary decisions, scalar judgments, borderline cases, scale points, granularity, central tendency bias, binary checks, construct, criteria, rubrics and guidelines
5.2.2 Absolute vs. relative judgments
This section compares relative and absolute judgments and explains when and how to use relative comparisons effectively.
- Advantages of relative judgments: Explain why comparing outputs can be easier and more sensitive than assigning absolute scores, and when that sensitivity may break down.
Why it matters: Comparisons can make judgments easier and reveal differences that scores miss, but not every preference is meaningful. - Comparison versus absolute standards: Distinguish situations suited to relative judgments from those requiring absolute judgments.
Why it matters: A system can win a comparison and still fail an important safety or quality requirement. - Comparison biases in judging: Explain how position bias and contrast bias can distort relative judgments by human judges.
Why it matters: A comparison can reflect presentation or the comparison set rather than the outputs' intrinsic quality. - Clear and manageable comparisons: Describe how to make relative judgment tasks clearer and less prone to noisy judgments.
Why it matters: Even a simple comparison needs a clear basis for deciding and a manageable task size.
Key terms: relative judgments, absolute score, calibration, agreement, absolute judgments, rankings, thresholds, position bias, contrast bias, comparison set
5.3 Defining rubrics and providing guidelines
This section defines evaluation rubrics and explains why their quality matters for consistent, focused judgments.
- Rubric purpose and scope: Describe what a rubric specifies and where it is used to acquire judgments.
Why it matters: A rubric gives judges a shared basis for turning observations into judgments. - Rubric quality and consistency: Explain how rubric quality affects the consistency and focus of evaluation judgments.
Why it matters: Poor instructions can make an evaluation inconsistent or cause it to measure the wrong thing.
Key terms: rubric, criteria, judgment, AI-as-a-Judge, evaluation dimension, personal interpretation, agreement between judges
5.3.1 Breaking constructs into observable criteria
This section explains how to turn abstract evaluation constructs into observable, well-defined rubric criteria.
- Observable construct criteria: Explain how to break an abstract evaluation construct into properties or behaviors that judges can inspect, using helpfulness as an example.
Why it matters: Observable criteria give judges something concrete to assess rather than relying on an intuitive impression. - Necessary and optional components: Distinguish criteria essential to a construct from optional enhancements, and explain why the distinction matters.
Why it matters: The distinction helps judges focus on what is required without treating every desirable feature as mandatory. - Failure mode checks: Describe how to turn common task failures into explicit checks that judges can verify.
Why it matters: Known failures can become specific checks instead of remaining vague concerns. - Independent non-overlapping criteria: Explain why criteria should be independent and non-overlapping, using clarity and coherence as an example.
Why it matters: Distinct criteria make judgments easier to apply without counting the same problem twice.
Key terms: construct, observable criteria, helpfulness, necessary components, optional components, failure modes, explicit checks, independent criteria, non-overlapping criteria, clarity
5.3.2 Giving examples and counterexamples
This section explains how different kinds of representative examples make evaluation rubrics clearer and more consistent for judges.
- Positive and negative anchors: Explain how prototypical positive examples and counterexamples clarify an evaluation criterion.
Why it matters: Clear anchors give judges a shared picture of what success and failure look like. - Borderline judgment thresholds: Explain how borderline examples help judges apply a rubric consistently.
Why it matters: Gray-area examples help judges make more consistent calls when a case is not obviously good or bad. - Contrastive dimension examples: Explain how contrastive examples isolate the dimension being judged.
Why it matters: Contrastive cases help judges avoid confusing the target criterion with other qualities. - Representative rubric examples: Explain why rubric examples should reflect the data judges will encounter.
Why it matters: Realistic variety prepares judges to apply the rubric to actual outputs, not just tidy examples.
Key terms: prototypical positive examples, prototypical counterexamples, construct, borderline or ambiguous examples, thresholds, contrastive examples, dimension, representative examples, subdomains, difficulty levels
5.3.3 Writing instructions that minimize ambiguity and vagueness
This section explains how to write clear judge instructions that reduce ambiguity and support consistent judgments.
- Operational task instructions: Explain how to define a judgment task in concrete terms, including what to evaluate, what to ignore, and the expected answer format.
Why it matters: Concrete task instructions make it easier for judges to apply a rubric consistently. - Permitted information boundaries: Explain why instructions should state what information judges may and may not use, using the medical chatbot example.
Why it matters: Judges can only apply the intended evidence standard if they know which information is allowed. - Misunderstanding and disagreement clarifications: Explain how to identify recurring judge misunderstandings and address them explicitly in instructions.
Why it matters: Clarifying recurring pitfalls helps judges make more consistent decisions. - Decision granularity expectations: Describe how instructions should guide judges when using binary versus scalar judgment formats.
Why it matters: Instructions about granularity help judges use the chosen answer format as intended. - Judgment workflow checklist: Describe how a short checklist can guide judges through a multistep judgment.
Why it matters: A shared sequence helps judges follow the same decision process.
Key terms: operational terms, faithfulness, retrieved passages, retrieved evidence, rubric, medical knowledge, pilot annotation, calibration, judge disagreements, borderline cases
5.4 Deciding who will judge
This section introduces why judge selection matters and identifies common judge types and hybrid setups.
- Judge selection and evaluation validity: Explain why the choice of judge should match the judgment task and what can happen when it does not.
Why it matters: Matching judges to the task helps make evaluation conclusions useful beyond the test. - Judge types and hybrids: Describe the four broad categories of judges and the option of combining judge types in a hybrid setup.
Why it matters: Knowing the available judge types makes it possible to consider a combination rather than relying on just one source.
Key terms: expertise, context, calibration, generalize, domain experts, trained non-experts, crowdsourced contributors, AI-based judges, hybrid setup
5.4.1 Domain experts
This section explains the value, limitations, and appropriate uses of domain experts in evaluation, and why their judgments can disagree.
- Expert judgment strengths and limitations: Explain what domain experts contribute to evaluation and why relying on them can be difficult at scale.
Why it matters: These trade-offs help you decide when expert judgment is worth its cost. - Expert investment and hybrid roles: Justify when to invest in domain experts and describe roles they can play alongside other judges.
Why it matters: Targeting expert involvement can put scarce expertise where it matters most. - Sources of expert disagreement: Explain why experts can disagree and what that disagreement can reveal.
Why it matters: Expert disagreement can reveal ambiguity instead of simply indicating a bad judgment.
Key terms: domain experts, technical competence, contextual awareness, rubrics, adjudicators, hybrid pipeline, tacit knowledge, subspecialization, contextual inference, scientific or professional uncertainty
5.4.2 Trained non-experts
This section explains when trained non-expert annotators are useful, their advantages and limitations, and the support needed for reliable judgments.
- Trained non-expert capabilities: Explain how trained non-experts can provide reliable judgments and what advantages they offer over domain experts.
Why it matters: Trained non-experts can make judgment collection more scalable without necessarily sacrificing consistency. - Training and quality limitations: Describe the preparation and monitoring non-expert judges need and the cases they may struggle to judge.
Why it matters: Lower recruitment costs do not eliminate the work of preparing and monitoring judges. - Suitable tasks and workflows: Explain when non-expert annotators are a good fit and how they can be used alongside experts or to pilot judgment pipelines.
Why it matters: Matching judges to task difficulty and stakes helps teams use expertise where it is most needed.
Key terms: trained non-expert annotators, rubrics, calibration, organizational standards, onboarding, quality checks, judgment drift, feedback loops, medium-stakes evaluations, hybrid workflows
5.4.3 Crowdsourced workers
This section explains when to use crowdsourced judges, their benefits and limitations, and ethical and legal responsibilities toward human judges.
- Crowdsourcing benefits and suitability: Explain how crowdsourced judging works, its advantages, and the kinds of tasks it suits.
Why it matters: Choosing crowdsourcing for the right tasks lets you benefit from its speed, scale, and range of perspectives. - Crowdsourcing quality and skew: Explain why crowdsourced judgments can be unreliable or unrepresentative and why quality control is necessary.
Why it matters: A large, inexpensive pool of judgments is not automatically reliable or representative. - Sensitive data protection: Describe the data-protection responsibilities involved when human judges encounter sensitive information.
Why it matters: Protecting sensitive data is a central obligation when people review evaluation material. - Fair working conditions: Describe fair and transparent working conditions for paid human judges.
Why it matters: Ethical evaluation protects the people doing the judging as well as the data. - Informed consent and transparency: Explain what judges need to understand to participate with informed consent.
Why it matters: Clear information lets judges make an informed choice about participating. - Psychological risk safeguards: Describe safeguards for judges who may encounter psychologically harmful content.
Why it matters: Judgment tasks can expose people to disturbing material that requires active safeguards.
Key terms: crowdsourcing, crowdworkers, subjective impressions, demographic skew, quality control, calibration, noise, data protection, anonymizing, pseudonymizing
5.4.4 AI-as-a-Judge
This section explains how AI judges produce evaluation judgments, their scalability benefits and limitations, and when human validation and review are needed.
- AI judge role: Explain what AI-as-a-judge does and how it can be instructed to provide judgments.
Why it matters: This establishes what an AI judge contributes to an evaluation. - Scalability and evaluation uses: Explain why AI judges are useful for large-volume evaluations and give an example of an enabled use.
Why it matters: Speed and cost make some evaluation workflows practical at a scale that would otherwise be too slow or expensive. - Sensitivity and judgment biases: Describe how changes in evaluation setup and model biases can distort AI judgments.
Why it matters: An AI judge can produce different or systematically misleading ratings for reasons unrelated to answer quality. - Human validation and oversight: Explain what evidence supports AI judging and why human validation and review remain necessary.
Why it matters: Human comparison and oversight help prevent promising automated signals from being treated as universally reliable.
Key terms: AI-as-a-judge, judgments, rubrics, scalability, continuous production monitoring, robustness sweeps, ablations, prompt phrasing, ordering of arguments, formatting choices
5.5 Calibrating human judges
This section distinguishes practice rounds for individual judge learning from calibration rounds for consistent group interpretation of a rubric.
- Individual judge practice: Describe how practice rounds help judges learn the task and rubric before full-scale judging, and give examples of activities that support learning and clarification.
Why it matters: Practice helps judges find and resolve misunderstandings before actual judging begins. - Group calibration consistency: Explain the purpose of calibration rounds and distinguish it from the purpose of practice rounds.
Why it matters: A shared interpretation makes judgments more consistent across people.
Key terms: practice rounds, rubric, gold judgment, calibration rounds, consistency across annotators
5.6 Developing and validating LLM judges
This section introduces a baseline LLM judge prompt and explains how judgment format can affect reliability.
- Baseline LLM judge prompt: Describe how to start building an LLM judge and how the baseline relates to the evaluation’s reliability needs.
Why it matters: A structured prompt gives you a working judge that you can improve to suit the evaluation. - Judgment format and reliability: Explain how judgment format can affect LLM judge reliability and how to choose a starting format.
Why it matters: Choosing distinctions the judge can apply clearly can improve the consistency of its judgments.
Key terms: LLM judge, rubric, judgment task, output format, reliability, judgment format, grading scale, binary judgments, numeric scoring, categorical labels
5.6.1 Which LLMs to use as judges
This section explains how to select an LLM judge based on specialization, model strength, and independence from the system being evaluated.
- Generalist and specialist judges: Explain when a general-purpose LLM is sufficient and when a specialized judge may be worth considering.
Why it matters: This helps you avoid unnecessary specialization while recognizing tasks that demand deep domain expertise. - Judge strength trade-offs: Explain the benefits of stronger judge models and the practical reasons a weaker model may be appropriate.
Why it matters: The best judge balances judgment quality against the demands and cost of the evaluation. - Starting strength strategies: Compare starting with a weaker judge and starting with a stronger judge when selecting model strength.
Why it matters: The starting model affects both evaluation cost and how easily you can diagnose problems with judging. - Self-judgment and independence: Describe when a model judging its own outputs may be useful and how an independent judge relates to self-bias and reliability.
Why it matters: Choosing an independent judge can reduce bias concerns, but all judges still need reliable evaluation.
Key terms: general-purpose LLM, specialized domain model, rubric, judge model strength, rubrics, expert evaluations, weaker model, starting strong, performance ceiling, prompts
5.6.2 Writing effective judgment prompts
This section explains how to write explicit LLM-judge prompts that produce more accurate, stable, and reproducible judgments.
- Explicit judgment instructions: Explain how to translate a rubric into explicit instructions for an LLM judge, including relevant rules and, where appropriate, a structured reasoning process.
Why it matters: Explicit instructions help the judge apply the intended criteria rather than inventing unstated rules. - Rigid and separated outputs: Describe how to constrain the judge's output format and separate the final decision from any explanation.
Why it matters: A stable decision field makes judgments easier to use without confusing them with commentary. - Permitted information sources: Explain why a judgment prompt should identify which information sources the model may and may not use.
Why it matters: A judgment can miss the intended criterion if the model draws on information the task does not permit. - Categorical judgment language: Describe how a prompt can discourage hedging by requesting a categorical judgment without probabilistic language or multiple labels.
Why it matters: A clear category is easier to interpret than a hedged or mixed verdict. - Bias and shortcut safeguards: Explain how to prompt the judge to evaluate examples independently and avoid superficial cues, guesswork, and partial evaluation.
Why it matters: These instructions help the judge focus on the evidence rather than presentation or speculation.
Key terms: rubric, judgment criteria, implicit assumptions, structured reasoning, output format, schema, decision, explanation, information sources, retrieved evidence
5.6.3 Validating and improving LLM judges
This section explains how to validate and iteratively improve an LLM judge against calibrated human judgments, and when to consider fine-tuning.
- Human Reference Validation: Describe how to set up a validation comparison between an LLM judge and human evaluators, and explain why humans serve as the reference.
Why it matters: A consistent comparison against human judgments provides a practical indication of whether the judge can be trusted. - Judgment Format Agreement Measures: Describe how agreement measures differ for binary, scalar, and pairwise judgments.
Why it matters: Choosing measures suited to the judgment format makes the human–LLM comparison meaningful. - Disagreement Driven Refinement: Trace the iterative process for improving an LLM judge when its agreement with humans is unsatisfactory.
Why it matters: Examining and retesting disagreements turns validation into a way to improve the judge. - Fine Tuning Decision: Explain why fine-tuning is not the first move and when it becomes appropriate for an LLM judge.
Why it matters: Starting with simpler refinements can avoid unnecessary training while leaving fine-tuning available for persistent gaps.
Key terms: calibrated human judges, validation dataset, rubric, rationales, precision, recall, F1 scores, confusion matrices, Spearman’s ρ, Kendall’s τ
5.6.4 Case study: Validating LLM judges for RAG faithfulness evaluation
This case study validates two RAG faithfulness judges against human-labeled inference examples and compares their performance across relationship types.
- Human Judgment Validation Design: Explain how the experiment used human-labeled SNLI examples to validate the two LLM judges, including how it controlled the comparison.
Why it matters: This design shows how human judgments can test an LLM judge before its results are trusted. - Toolkit Faithfulness Definitions: Distinguish the two toolkits' definitions of faithfulness and explain how they affect the expected judgment for a neutral example.
Why it matters: A fair validation must compare each judge against expectations that match its own definition of faithfulness. - Relationship Type Results: Describe how the judges' results differed on entailment, neutral, and contradiction cases, and what those differences suggest.
Why it matters: The results show why overall trust in a judge should be informed by performance on different kinds of cases.
Key terms: SNLI, premise, hypothesis, ground truth, classification, F1, faithfulness, neutral, entailment, contradiction
5.7 Designing judgment workflows
Defines a judgment workflow as the end-to-end process for producing and maintaining evaluation judgments.
- Judgment workflow scope: Describe what a judgment workflow covers, from assigning items to judges through handling assessments and maintaining quality.
Why it matters: This establishes the scope of the process that turns individual assessments into evaluation judgments.
Key terms: judgment workflow, judges, assessments, aggregating, monitoring quality
5.7.1 Number and allocation of judges
This section explains how to choose the number of judges per item and allocate judgment effort according to cost, reliability, and case difficulty.
- Single and multiple judges: Explain the trade-offs between assigning one judge and multiple judges to each item, including why independent errors matter.
Why it matters: The number of judges affects both workflow cost and protection against mistakes. - Pilot-based judge count: Describe how to use a pilot batch and a higher-confidence reference to choose the number of judges.
Why it matters: A pilot helps avoid paying for extra judges that add little useful reliability. - Stratified judge allocation: Explain how stratified allocation assigns judges to predefined groups and what it depends on.
Why it matters: Predefined routing can reserve expert review for cases that need it most. - Adaptive judge allocation: Describe how adaptive allocation escalates cases using available signals and explain its routing risk.
Why it matters: Dynamic escalation can focus review effort, but only on cases its signals successfully flag.
Key terms: single-judge workflows, multiple judges, independent errors, biases, pilot batch, higher-confidence reference, expert adjudication, majority vote, diminishing returns, stratified allocation
5.7.2 Order and sequence of judgments
This section compares parallel, sequential, and hybrid workflows for multiple judges evaluating the same item.
- Fully parallel judgment workflows: Explain how fully parallel judging works and its trade-off between independence and efficiency.
Why it matters: Independent judgments make agreement easier to interpret, but they require more judging effort. - Fully sequential judgment workflows: Explain how sequential judging works and why it can save effort while exposing later judges to influence.
Why it matters: Sequential review can reduce judging effort, but earlier decisions may shape later assessments. - Hybrid review selection strategies: Describe how a hybrid workflow combines independent judgments with selective review, and distinguish disagreement-based escalation from sampling.
Why it matters: Selective review focuses effort while sampling can catch shared mistakes that agreement alone would miss.
Key terms: fully parallel workflow, inter-rater agreement, aggregation, fully sequential workflows, anchoring, conformity effects, parallel and sequential elements, disagreement, sampling, adjudicator
5.7.3 Judgment aggregations
This section explains how to aggregate multiple judgments and respond to disagreement at the item or evaluation-process level.
- Judgment aggregation strategies: Explain how judgment format and trust in judges affect the choice of aggregation strategy.
Why it matters: The aggregation method should reflect both the kind of judgment and how much each judge is trusted. - Disagreement signals and thresholds: Explain why disagreement can be informative and how tolerance rules distinguish routine variation from disagreement that warrants attention.
Why it matters: Treating all disagreement as noise can hide problems in the evaluation. - Responses to elevated disagreement: Distinguish adjudication for an individual disputed item from structured analysis of recurring disagreement across items.
Why it matters: The right response depends on whether disagreement is isolated or points to a recurring evaluation problem.
Key terms: majority voting, mean, median, weighted aggregation, probabilistic aggregation models, disagreement thresholds, tolerance rules, score spread, adjudication, structured disagreement analysis
5.7.4 Judgment quality monitoring
This section explains why judgment quality requires active monitoring over time.
- Ongoing judgment quality monitoring: Explain why judgment quality needs active monitoring over time, including how problems can develop without an obvious individual error.
Why it matters: Monitoring helps detect gradual changes that might otherwise undermine judgment quality.
Key terms: judgment quality, active monitoring, drift, rubric updates
5.8 References
This section lists references cited in the chapter on acquiring judgments.
5.9 Summary
Concepts explained in this chapter
- LLM-as-a-judgeAn AI system that assesses outputs or behavior according to evaluation criteria.
- binary judgmentA judgment that classifies an output as either meeting a criterion or not, with no intermediate states.
- judgment formatA form in which a judgment is expressed, such as a binary choice, a scalar value, or a comparison between outputs.
- absolute judgmentA method of assessment in which a judge evaluates an output independently, such as by assigning it a score on a scale.
- relative judgmentA method of assessment in which judges compare outputs with one another rather than assigning each an independent score.
- scalar judgmentA judgment that assigns an output a value on a numerical or ordinal scale.
- evaluation criterionA stated condition or standard used to determine whether a system satisfies a decision requirement.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
