liveBook · Manning

Outcome-only supervision

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Outcome supervision trains or evaluates a model using the correctness or quality of its final result without directly supervising intermediate steps.

Outcome-only supervision trains or evaluates a model with one scalar judgment of its final result. It can identify that an answer is bad, but not where a reasoning trace failed. Outcome reward models (ORMs) apply this pattern at the end of a response. section 7.7.1

Use the contrast with process supervision to identify what the signal can and cannot support:

  • Outcome supervision scores only the final answer, making it cheaper than labeling every intermediate reasoning step. section 7.7.1
  • Process supervision scores each step as correct, incorrect, or neutral, providing denser feedback for credit assignment. section 7.7.1
  • Outcome supervision cannot distinguish a correct answer reached through faulty reasoning from one reached through correct reasoning. section 7.7.1
  • Longer responses worsen its credit-assignment problem, while reward hacking and other RLHF failure modes remain possible. section 7.8

Five-step math problem

An outcome reward assigns one score after the final answer. If that answer is wrong, the model learns only that the complete response failed. A process reward model can mark steps 1 and 2 correct, identify step 3 as the error, and expose why later steps went wrong. section 7.7.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains scalar final scoring, contrasts step-level feedback, and stops short of specifying a training algorithm.
Examples
Uses a five-step math trace to show how final-answer scoring hides the first incorrect reasoning step.
What you can do after
Evaluate whether final-result rewards provide enough diagnostic signal for reasoning-heavy tasks.
How it connects
Connects outcome reward models to process supervision, credit assignment, RLHF, reward hacking, and verifiable rewards.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.