A value head estimates the expected reward for a language model’s generated response. In PPO fine-tuning, it usually shares the transformer backbone with the policy model, which predicts next-token probabilities. section 10.8.5
Single-turn chatbot
For a single-turn response, the return is the reward and the advantage is reward minus the value estimate. Multi-turn dialogue uses Generalized Advantage Estimation instead. section 10.8.5
Listing 10.6 Advantage Computation
def compute_advantages_simple(
rewards: torch.Tensor,
values: torch.Tensor,
gamma: float = 1.0,
gae_lambda: float = 0.95,
) -> Tuple[torch.Tensor, torch.Tensor]:
advantages = rewards - values #A
returns = rewards #B
advantages = (advantages –
➥ advantages.mean()) /
➥(advantages.std() + 1e-8) #C
return advantages, returnsSee the simple advantage calculation: subtract values from rewards, then use rewards as returns. section 10.8.5
