liveBook · Manning

Group-relative baseline

As explained in Applied Reinforcement Learning

A baseline that compares a response’s reward with the rewards of other responses generated for the same prompt.

A group-relative baseline judges each answer against other answers generated for the same prompt. This avoids a separate value head for estimating typical reward, a challenging estimate for long-form generation. section 11.4

For one prompt, the procedure is:

  • Generate G responses, using sampling so they can differ. section 11.4
  • Score each response with the verifier, then calculate the group mean μG and standard deviation σG. section 11.4
  • Normalize each reward as Âi = (R(yi) − μG) / σG. Above-average answers get positive advantages; below-average answers get negative ones. section 11.4

When every answer is wrong

If all responses receive the same reward, the group standard deviation is zero and all advantages are zero. section 11.4

How Applied Reinforcement Learning covers it

How deep it goes
Calculates normalized advantages from group rewards and explains their signs and the equal-reward case.
Examples
Uses sampled answers to one prompt, including a group whose responses all receive the same reward.
What it assumes
Assumes responses can be sampled for a prompt and scored with a verifier.
What you can do after
Understand how an answer’s reward compares with its prompt-specific group and determines advantage direction.
How it connects
Connects PPO, value heads, reward-based advantages, and reinforcement learning with verifiable rewards.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.