Absolute judgment assigns a standalone score to one response against defined criteria, rather than comparing it with another response. Scoring a summary from 1–5 for coherence is an absolute judgment; choosing the better of two summaries is a relative, pairwise judgment. Absolute scoring is generally more difficult because the evaluator must interpret the scale consistently without a comparison item. section 4.2.2
Make an absolute judgment more reproducible by:
- Defining the evaluation dimension precisely, such as coherence. section 4.2.2
- Adding concrete guidance about structure and organization instead of relying on an undefined quality impression. section 4.2.2
- Using explicit evaluation steps and a final score when implementing structured LLM judging. section 4.2.2
Coherence scoring
Ask an evaluator to rate one news summary from 1 to 5 for coherence, judging whether its sentences form a well-organized body of information rather than a heap of related facts. section 4.2.2

