liveBook · Manning

Chain-of-thought

As explained in AI Model Evaluation

The process of deriving conclusions or solutions through logical or evidential steps expressed in a model output.

In LLM-as-a-judge evaluation, chain-of-thought means asking a judge to work through rubric criteria explicitly before returning a score. The resulting reasoning makes the basis of a verdict inspectable; a bare number does not. section 10.3.1

A reasoning-first scoring prompt has practical trade-offs:

  • Require criterion-level reasoning before the score so the judge engages with the evidence named by the rubric. section 10.3.1
  • Inspect the reasoning to diagnose whether a wrong score came from misunderstanding the task, missing context, or applying the rubric differently. section 10.3.1
  • Budget for extra tokens and latency; a fluent explanation does not prove the judgment is correct. section 10.3.1

Bare score versus reasoning first

A prompt such as “rate this response 1–5” allows a surface-impression verdict. Asking for a short justification for each rubric dimension before the score leaves reasoning to inspect when the score looks wrong. section 10.3.1

How AI Model Evaluation covers it

How deep it goes
Explains criterion-by-criterion reasoning and its debugging value, but cautions that fluent explanations do not prove correctness.
Examples
Contrasts a bare 1–5 rating prompt with criterion justifications requested before the score.
What it assumes
Assumes a scoring rubric whose criteria the judge can consider before assigning a score.
What you can do after
Diagnose questionable judge scores by inspecting misunderstandings, missed context, or rubric application.
How it connects
Connects reasoning traces to evaluation rubrics, scoring prompts, and LLM-as-a-judge reliability.

Also explained in

Compare how 21 books explain chain-of-thought →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.