liveBook · Manning

Accuracy

As explained in Reinforcement Learning from Human Feedback

Training accuracy is the proportion of training examples for which a model produces the correct output.

Correctness measures whether a model gives the right answer for a task with a determinate target. Training accuracy expresses this as the proportion of training examples classified correctly, but it is a measurement rather than a guarantee of real-world quality. section 10.1 section 14.1.1

Separate correctness from preference-based evaluation:

  • Use correctness when a response has a clear right or wrong answer, such as identifying a president. section 10.1
  • Use preference judgments when alternatives may all be correct or incorrect and must be compared for clarity, safety, helpfulness, or degree of error. section 11.2
  • Interpret training accuracy separately from held-out accuracy: the former can rise while the latter falls through memorization, whereas proxy over-optimization changes the objective itself. section 14.1.1

Why correctness is not enough

A factual question can have a clear correct answer, while two poems may be judged by preference rather than correctness. Likewise, a response can maximize a reward model through verbosity or rare-token repetition without becoming more helpful. section 10.1 section 14.1.1

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Distinguishes correctness, training accuracy, held-out accuracy, and proxy over-optimization without developing an implementation procedure.
Examples
Uses factual questions, creative poems, verbose answers, and rare-token repetition to contrast correctness with preference and proxy scores.
What you can do after
Evaluate/understand whether an observed score measures exact task correctness, generalization, or a potentially misaligned proxy.
How it connects
Connects correctness to preference data, reward models, training accuracy, held-out accuracy, overfitting, and over-optimization.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Also explained in

Compare how 32 books explain accuracy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.