A group-relative baseline judges each answer against other answers generated for the same prompt. This avoids a separate value head for estimating typical reward, a challenging estimate for long-form generation. section 11.4
For one prompt, the procedure is:
- Generate G responses, using sampling so they can differ. section 11.4
- Score each response with the verifier, then calculate the group mean μG and standard deviation σG. section 11.4
- Normalize each reward as Âi = (R(yi) − μG) / σG. Above-average answers get positive advantages; below-average answers get negative ones. section 11.4
When every answer is wrong
If all responses receive the same reward, the group standard deviation is zero and all advantages are zero. section 11.4
