An advantage function is the policy-gradient signal for whether an action was better or worse than expected at its state. Instead of reinforcing raw return alone, it subtracts a baseline: Aπ(s,a) = Qπ(s,a) − Vπ(s). This comparison reduces noisy gradient estimates while preserving an unbiased expected gradient. Positive advantages increase an action’s probability, negative advantages decrease it, and zero advantage produces no policy update. section 6.2.1 section 6.2.2
Equation 6.21
\[\label{eq:advantage_policy_gradient} \nabla_\theta J(\theta) = \mathbb{E}_\tau \left[ \sum_{t=0}^T \nabla_\theta \log \pi_\theta(a_t|s_t) A^{\pi_\theta}(s_t, a_t) \right].\]The advantage-weighted policy gradient replaces the raw return with a relative signal at every action. The log-probability gradient assigns credit to the sampled action; the advantage determines its direction and strength. section 6.2.2
For a learned critic, the practical estimate is Aₜ = Gₜ − V(sₜ): the observed or computed return minus the value model’s predicted remaining return. Equivalently, a one-step form is the temporal-difference residual rₜ + γV(sₜ₊₁) − V(sₜ). PPO uses such per-token advantages, often with generalized advantage estimation, while training the value function against rollout-derived targets. section 6.2.1 section 6.2.7
Advantage estimation choices trade memory, sampling, and credit assignment:
- A simple batch or moving-average baseline normalizes returns without depending on the sampled action, reducing variance. section 6.2.2
- RLOO samples multiple completions for each prompt and subtracts the mean reward of the other completions. It needs no value network, remains exactly unbiased, and gives every token in a completion one sequence-level advantage. section 6.2.4
- GRPO normalizes each completion reward by its prompt group’s mean and standard deviation. Like RLOO, it compares peer completions and broadcasts a sequence-level signal, but commonly applies KL regularization directly in the loss. section 6.2.8
- PPO uses a learned value function to assign distinct token-level advantages, at the cost of an additional model and value-function training. section 6.2.7
Two completions, one prompt
If two completions receive rewards 3 and 1, a batch-mean baseline of 2 yields sequence advantages +1 and −1. The positive completion is reinforced and the negative completion is discouraged. In a bandit-style implementation, each scalar is expanded across all tokens of its completion; an MDP-style value model instead can produce a different advantage at each token. section 6.3.2 section 6.3.1
The choice is not merely mathematical. Sequence-level, group-relative estimates suit settings with multiple sampled answers per prompt and avoid a critic. Per-token estimates offer finer credit assignment but require value predictions and associated training. In either case, loss aggregation, KL placement, clipping, and reward sparsity affect stability and the final policy. section 6.3.2 section 6.6
