liveBook · Manning

Chapter 5: Acquiring judgments

What you will learn in chapter 5 of Evaluating AI Systems

This introduction defines judgments, explains their role in evaluation metrics, and shows why judgment quality affects evaluation results.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction defines judgments, explains their role in evaluation metrics, and shows why judgment quality affects evaluation results.

Key terms: judgment, evaluation criterion, label, rating, ranking, preference, judgment-based metrics, reference-based metrics, reference answers, gold labels

5.1 “Who and how decides what’s correct?”

The section introduces judgment-acquisition trade-offs for evaluating a medical advice chatbot.

Key terms: judgments, binary labels, correct, safe, domain experts, trained non-experts, rubric, redundancy, disagreement, LLM as a judge

5.2 Deciding what judgments should look like

This section introduces how evaluation judgments are shaped by independent choices between binary or scalar and absolute or relative formats.

Key terms: judgment format, metrics, custom metrics, binary judgments, scalar judgments, numerical scale, ordinal scale, absolute judgments, relative judgments, pairwise comparisons

5.2.1 Binary vs. scalar judgments

This section compares binary and scalar judgments and presents structured binary checks as an alternative to subjective overall ratings.

Key terms: binary decisions, scalar judgments, borderline cases, scale points, granularity, central tendency bias, binary checks, construct, criteria, rubrics and guidelines

5.2.2 Absolute vs. relative judgments

This section compares relative and absolute judgments and explains when and how to use relative comparisons effectively.

Key terms: relative judgments, absolute score, calibration, agreement, absolute judgments, rankings, thresholds, position bias, contrast bias, comparison set

5.3 Defining rubrics and providing guidelines

This section defines evaluation rubrics and explains why their quality matters for consistent, focused judgments.

Key terms: rubric, criteria, judgment, AI-as-a-Judge, evaluation dimension, personal interpretation, agreement between judges

5.3.1 Breaking constructs into observable criteria

This section explains how to turn abstract evaluation constructs into observable, well-defined rubric criteria.

Key terms: construct, observable criteria, helpfulness, necessary components, optional components, failure modes, explicit checks, independent criteria, non-overlapping criteria, clarity

5.3.2 Giving examples and counterexamples

This section explains how different kinds of representative examples make evaluation rubrics clearer and more consistent for judges.

Key terms: prototypical positive examples, prototypical counterexamples, construct, borderline or ambiguous examples, thresholds, contrastive examples, dimension, representative examples, subdomains, difficulty levels

5.3.3 Writing instructions that minimize ambiguity and vagueness

This section explains how to write clear judge instructions that reduce ambiguity and support consistent judgments.

Key terms: operational terms, faithfulness, retrieved passages, retrieved evidence, rubric, medical knowledge, pilot annotation, calibration, judge disagreements, borderline cases

5.4 Deciding who will judge

This section introduces why judge selection matters and identifies common judge types and hybrid setups.

Key terms: expertise, context, calibration, generalize, domain experts, trained non-experts, crowdsourced contributors, AI-based judges, hybrid setup

5.4.1 Domain experts

This section explains the value, limitations, and appropriate uses of domain experts in evaluation, and why their judgments can disagree.

Key terms: domain experts, technical competence, contextual awareness, rubrics, adjudicators, hybrid pipeline, tacit knowledge, subspecialization, contextual inference, scientific or professional uncertainty

5.4.2 Trained non-experts

This section explains when trained non-expert annotators are useful, their advantages and limitations, and the support needed for reliable judgments.

Key terms: trained non-expert annotators, rubrics, calibration, organizational standards, onboarding, quality checks, judgment drift, feedback loops, medium-stakes evaluations, hybrid workflows

5.4.3 Crowdsourced workers

This section explains when to use crowdsourced judges, their benefits and limitations, and ethical and legal responsibilities toward human judges.

Key terms: crowdsourcing, crowdworkers, subjective impressions, demographic skew, quality control, calibration, noise, data protection, anonymizing, pseudonymizing

5.4.4 AI-as-a-Judge

This section explains how AI judges produce evaluation judgments, their scalability benefits and limitations, and when human validation and review are needed.

Key terms: AI-as-a-judge, judgments, rubrics, scalability, continuous production monitoring, robustness sweeps, ablations, prompt phrasing, ordering of arguments, formatting choices

5.5 Calibrating human judges

This section distinguishes practice rounds for individual judge learning from calibration rounds for consistent group interpretation of a rubric.

Key terms: practice rounds, rubric, gold judgment, calibration rounds, consistency across annotators

5.6 Developing and validating LLM judges

This section introduces a baseline LLM judge prompt and explains how judgment format can affect reliability.

Key terms: LLM judge, rubric, judgment task, output format, reliability, judgment format, grading scale, binary judgments, numeric scoring, categorical labels

5.6.1 Which LLMs to use as judges

This section explains how to select an LLM judge based on specialization, model strength, and independence from the system being evaluated.

Key terms: general-purpose LLM, specialized domain model, rubric, judge model strength, rubrics, expert evaluations, weaker model, starting strong, performance ceiling, prompts

5.6.2 Writing effective judgment prompts

This section explains how to write explicit LLM-judge prompts that produce more accurate, stable, and reproducible judgments.

Key terms: rubric, judgment criteria, implicit assumptions, structured reasoning, output format, schema, decision, explanation, information sources, retrieved evidence

5.6.3 Validating and improving LLM judges

This section explains how to validate and iteratively improve an LLM judge against calibrated human judgments, and when to consider fine-tuning.

Key terms: calibrated human judges, validation dataset, rubric, rationales, precision, recall, F1 scores, confusion matrices, Spearman’s ρ, Kendall’s τ

5.6.4 Case study: Validating LLM judges for RAG faithfulness evaluation

This case study validates two RAG faithfulness judges against human-labeled inference examples and compares their performance across relationship types.

Key terms: SNLI, premise, hypothesis, ground truth, classification, F1, faithfulness, neutral, entailment, contradiction

5.7 Designing judgment workflows

Defines a judgment workflow as the end-to-end process for producing and maintaining evaluation judgments.

Key terms: judgment workflow, judges, assessments, aggregating, monitoring quality

5.7.1 Number and allocation of judges

This section explains how to choose the number of judges per item and allocate judgment effort according to cost, reliability, and case difficulty.

Key terms: single-judge workflows, multiple judges, independent errors, biases, pilot batch, higher-confidence reference, expert adjudication, majority vote, diminishing returns, stratified allocation

5.7.2 Order and sequence of judgments

This section compares parallel, sequential, and hybrid workflows for multiple judges evaluating the same item.

Key terms: fully parallel workflow, inter-rater agreement, aggregation, fully sequential workflows, anchoring, conformity effects, parallel and sequential elements, disagreement, sampling, adjudicator

5.7.3 Judgment aggregations

This section explains how to aggregate multiple judgments and respond to disagreement at the item or evaluation-process level.

Key terms: majority voting, mean, median, weighted aggregation, probabilistic aggregation models, disagreement thresholds, tolerance rules, score spread, adjudication, structured disagreement analysis

5.7.4 Judgment quality monitoring

This section explains why judgment quality requires active monitoring over time.

Key terms: judgment quality, active monitoring, drift, rubric updates

5.8 References

This section lists references cited in the chapter on acquiring judgments.

5.9 Summary

Concepts explained in this chapter

← Chapter 4: Acquiring evaluation data

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.