liveBook · Manning

Chain-of-thought

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Reasoning ability is the capacity of a model to derive conclusions through logically appropriate intermediate inferences.

Multi-step reasoning is a model’s ability to derive conclusions through logically appropriate intermediate inferences. Evaluation must distinguish solving a problem from judging whether another solution reasons correctly: a judge may solve an elementary problem independently yet accept a contaminated, wrong answer. section 4.3.4

Treat multi-step reasoning as both a capability and an evaluation problem:

  • Classify tasks by whether correctness is mechanically verifiable or requires judgment about reasoning quality. section 4.3.4
  • Separate final-answer correctness from process correctness; an accidentally correct answer can conceal flawed intermediate steps. section 7.7.2
  • Avoid assuming that chain-of-thought or few-shot prompting creates independent verification, because evaluation context can contaminate the judge’s reasoning. section 4.3.4
  • Use step-level supervision when the reasoning path matters: label each step positive, neutral, or negative, then score candidate traces. section 7.7.2

Reference-guided grading

For a math or reasoning task, generate the judge’s solution independently before inspecting candidate answers. Compare the candidates against that reference instead of letting a plausible but incorrect rationale determine the evaluation. This reduces context contamination and improves grading accuracy. section 4.3.4

Process-supervised reranking

Generate many candidate solutions, score each reasoning step with a process reward model, and retain the highest-scoring candidate. On MATH, process-supervised reranking reached about 78% accuracy at large candidate-pool sizes, compared with about 72% for outcome supervision and 65% for majority voting. section 7.7.2

Figure 4.10 LLM judges can be misled by context even for problems within their capabilities. The presence of a wrong answer in the evaluation context can contaminate the judge's own reasoning.

Use the figure to inspect how an incorrect answer in the evaluation context can mislead a judge, even when the underlying problem is within the judge’s capabilities. section 4.3.4

Figure 4.11 Reference-guided grading generates the judge's solution independently before evaluation. This prevents context contamination and dramatically improves accuracy on math and reasoning tasks.

Use the figure to compare contaminated grading with reference-guided grading, which generates an independent solution before evaluating responses. section 4.3.4

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains context contamination, independent references, step-level labels, and candidate reranking, but stops short of a general reasoning algorithm.
Examples
Uses contaminated elementary math judgments, reference-guided grading, PRM800K step labels, and MATH candidate reranking.
What it assumes
Assumes basic familiarity with model outputs and evaluation, while teaching the distinction between final-answer and intermediate-step correctness.
What you can do after
Diagnose reasoning-evaluation failures and apply independent references or process reward models to rerank candidate solutions.
How it connects
Connects multi-step reasoning to LLM-as-a-judge, chain-of-thought prompting, reward models, process supervision, outcome supervision, and reinforcement learning.

Also explained in

Compare how 21 books explain chain-of-thought →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.