liveBook · Manning

Value head

As explained in Applied Reinforcement Learning

A language-model value head estimates the expected reward for a response generated by a language model.

A value head estimates the expected reward for a language model’s generated response. In PPO fine-tuning, it usually shares the transformer backbone with the policy model, which predicts next-token probabilities. section 10.8.5

Single-turn chatbot

For a single-turn response, the return is the reward and the advantage is reward minus the value estimate. Multi-turn dialogue uses Generalized Advantage Estimation instead. section 10.8.5

Listing 10.6 Advantage Computation
def compute_advantages_simple(
    rewards: torch.Tensor,
    values: torch.Tensor,
    gamma: float = 1.0,
    gae_lambda: float = 0.95,
    ) -> Tuple[torch.Tensor, torch.Tensor]:
    advantages = rewards - values  #A
    returns = rewards  #B
    advantages = (advantages –
    ➥ advantages.mean()) / 
    ➥(advantages.std() + 1e-8)  #C
    return advantages, returns

See the simple advantage calculation: subtract values from rewards, then use rewards as returns. section 10.8.5

How Applied Reinforcement Learning covers it

How deep it goes
Defines the estimate and its role in single-turn advantage calculation, while deferring multi-turn estimation to Generalized Advantage Estimation.
Examples
Uses a single-turn chatbot where the advantage is reward minus value and the return equals reward.
What you can do after
Understand how a value estimate contributes to advantage calculation during PPO fine-tuning.
How it connects
Connects the policy model, reward model, reference model, PPO, and advantage estimation.

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.