A value function estimates the expected return from a partial response state. In PPO, a learned value model predicts how promising the current state is, while the policy selects the next token. section 7.3.1 section 7.3.2
For rollout state sₖ, the value model Vϕ(sₖ) is trained from eventual returns. The resulting baseline converts return into an advantage: Aₖ = Rₖ − Vϕ(sₖ). section 7.3.2
A PPO-style value path typically adds:
- A value head or separate critic with parameters ϕ, conceptually distinct from policy parameters θ. section 7.3.2
- Value forward passes, a value loss, an optimizer path, and sometimes generalized advantage estimation. section 7.10.3
- Additional memory, computation, and engineering complexity, which become significant for large language models. section 7.3.2
Why GRPO removes the value function
GRPO samples several completions for one prompt and normalizes their rewards. If four completions receive rewards 1, 1, 0, and 0, the group mean is 0.5; correct completions receive positive advantages and incorrect ones negative advantages. Group statistics replace the learned value baseline, while the reference model still provides KL control. section 7.4 section 7.4.3
Use the equation to connect rollout returns, learned state values, and policy-gradient advantages. section 7.3.2



