chapter five

5 Acquiring judgments

 

This chapter covers

  • Determining the most appropriate format for your evaluation judgments
  • Defining effective rubrics and judgment guidelines
  • Selecting the optimal mix of judges for your evaluation context
  • Developing and validating LLM-based judges
“Wherever there is judgment, there is noise – and probably a lot more of it than you would think.”

—Daniel Kahneman

As we saw in Chapter 3, many evaluation metrics depend on judgments; judgment-based metrics rely on humans or AI judges to assess examples directly, while reference-based metrics often require judgments up front to create the reference answers, gold labels, or benchmark datasets they use as input.

In the context of AI evaluation, a judgment is an assessment that determines whether something satisfies a particular evaluation criterion. It may take the form of a label, rating, ranking, preference, or decision. Depending on the task, a judgment might be as simple as assigning a class label (“pneumonia” vs. “normal”) or as nuanced as rating a chatbot response for factual accuracy, bedside manner, and safety. For example, a radiologist may label a chest X-ray as showing pneumonia or not, an expert may rate a chatbot response, or a user may express a preference between two alternative responses.

5.1 “Who and how decides what’s correct?”

5.2 Deciding what judgments should look like

5.2.1 Binary vs. scalar judgments

5.2.2 Absolute vs. relative judgments

5.3 Defining rubrics and providing guidelines

5.3.1 Breaking constructs into observable criteria

5.3.2 Giving examples and counterexamples

5.3.3 Writing instructions that minimize ambiguity and vagueness

5.4 Deciding who will judge

5.4.1 Domain experts

5.4.2 Trained non-experts

5.4.3 Crowdsourced workers

5.4.4 AI-as-a-Judge

5.5 Calibrating human judges

5.6 Developing and validating LLM judges

5.6.1 Which LLMs to use as judges

5.6.2 Writing effective judgment prompts

5.6.3 Validating and improving LLM judges

5.6.4 Case study: Validating LLM judges for RAG faithfulness evaluation

5.7 Designing judgment workflows

5.7.1 Number and allocation of judges

5.7.2 Order and sequence of judgments

5.7.3 Judgment aggregations

5.7.4 Judgment quality monitoring

5.8 References

5.9 Summary