liveBook · Manning

Generative reward model

As explained in Reinforcement Learning from Human Feedback

A generative reward model is a reward model that generates evaluative judgments or preference assessments rather than only emitting a scalar score.

A generative reward model uses a prompted language model to judge two completions and generate an evaluative explanation or preference, rather than emitting only a scalar reward. This approach extends LLM-as-a-judge workflows into preference-data creation and evaluation, offering cheaper data collection while remaining less effective than established reward models on many reward-model evaluations. section 5.7 section 5.9

A typical generative judging workflow contains these operational choices:

  • Prompt the model with judging instructions, a user prompt, and two labeled completions; request a comparison, explanation, and final verdict. section 5.7
  • Use sampling temperature 0 when reducing rating variance is more important than sampling diversity. section 5.7

Principle-conditioned preference selection

A feedback model receives a prompt, selected constitutional principles, and two prior completions. It explains which answer better satisfies the stated principles, then selects answer A or B. section 12.5

[System]
Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below.
You should choose the assistant that follows the user's instructions and answers the user's question better.
Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses.
Begin your evaluation by comparing the two responses and provide a short explanation.
Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision.
Do not allow the length of the responses to influence your evaluation.
Do not favor certain names of the assistants.
Be as objective as possible.
After providing your explanation, output your final verdict by strictly following this format: "[[A]]" if assistant A is better, "[[B]]" if assistant B is better, and "[[C]]" for a tie.
[User Question]
{question}
[The Start of Assistant A's Answer]
{answer_a}
[The End of Assistant A's Answer]
[The Start of Assistant B's Answer]
{answer_b}
[The End of Assistant B's Answer]

Inspect a complete judge prompt that specifies evaluation criteria, bias safeguards, comparative reasoning, and a strict final-verdict format. section 5.7

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Defines the prompting workflow and practical variance control, but stops short of replacing scalar reward-model training internals.
Examples
Uses MT-Bench-style judging prompts and principle-conditioned A-versus-B comparisons from Constitutional AI.
What you can do after
Apply prompted language models to compare completions, produce explanations, and create preference judgments.
How it connects
Connects LLM-as-a-judge, preference data, reward modeling, RLHF, and Constitutional AI.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.