liveBook · Manning

Absolute judgment

As explained in LLM Evaluation and Alignment, The Foundational Ideas

An absolute judgment assigns a standalone quality or acceptability score to one response without comparing it with another response.

Absolute judgment assigns a standalone score to one response against defined criteria, rather than comparing it with another response. Scoring a summary from 1–5 for coherence is an absolute judgment; choosing the better of two summaries is a relative, pairwise judgment. Absolute scoring is generally more difficult because the evaluator must interpret the scale consistently without a comparison item. section 4.2.2

Make an absolute judgment more reproducible by:

  • Defining the evaluation dimension precisely, such as coherence. section 4.2.2
  • Adding concrete guidance about structure and organization instead of relying on an undefined quality impression. section 4.2.2
  • Using explicit evaluation steps and a final score when implementing structured LLM judging. section 4.2.2

Coherence scoring

Ask an evaluator to rate one news summary from 1 to 5 for coherence, judging whether its sentences form a well-organized body of information rather than a heap of related facts. section 4.2.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines the contrast with pairwise judgment and stresses clearer rubrics, while deferring detailed scoring mechanics.
Examples
Uses 1–5 coherence ratings for news summaries and structured reasoning steps as concrete evaluation artifacts.
What it assumes
Requires a defined evaluation metric and explicit rubric guidance for the quality dimension being scored.
What you can do after
Apply G-Eval-style scoring by defining criteria, generating evaluation steps, and producing a final assessment.
How it connects
Connects absolute judgment with scoring, pairwise preference, evaluation criteria, rubrics, and G-Eval.

Also explained in

Compare how 2 books explain absolute judgment →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.