5 Acquiring judgments
This chapter covers
- Determining the most appropriate format for your evaluation judgments
- Defining effective rubrics and judgment guidelines
- Selecting the optimal mix of judges for your evaluation context
- Developing and validating LLM-based judges
“Wherever there is judgment, there is noise – and probably a lot more of it than you would think.”
—Daniel Kahneman
As we saw in Chapter 3, many evaluation metrics depend on judgments; judgment-based metrics rely on humans or AI judges to assess examples directly, while reference-based metrics often require judgments up front to create the reference answers, gold labels, or benchmark datasets they use as input.
In the context of AI evaluation, a judgment is an assessment that determines whether something satisfies a particular evaluation criterion. It may take the form of a label, rating, ranking, preference, or decision. Depending on the task, a judgment might be as simple as assigning a class label (“pneumonia” vs. “normal”) or as nuanced as rating a chatbot response for factual accuracy, bedside manner, and safety. For example, a radiologist may label a chest X-ray as showing pneumonia or not, an expert may rate a chatbot response, or a user may express a preference between two alternative responses.