Value function

As explained in Reinforcement Learning from Human Feedback

A value network is a neural network that estimates the expected value of states or actions in reinforcement learning.

A value function estimates how much reward remains from a state. For a state s at time t, it is the expected discounted return: V(s) = E[G_t | S_t = s]. The discount γ balances near- and long-term rewards: γ = 0 keeps only immediate reward, while γ = 1 weights future rewards equally. In language-model RL, a state is a prompt plus the generated prefix, and each next token is an action. section 6.2 section 5.6.1

The estimate is policy-dependent: V^π(s) predicts return when continuing with policy π. Policy-gradient objectives weight state values by how often the current policy visits them, but implementations estimate this from sampled on-policy rollouts rather than enumerating states. Thus a value prediction must track a changing policy and changing rollout targets. section 6.2 section 5.6

In actor–critic training, the value network is the critic: an additional model copy or scalar regression head that predicts V_t at every completion token. Its main job is to provide a baseline for an advantage estimate, A_t = R̂_t − V_t. The advantage says whether observed return was better or worse than the critic expected, reducing variance in the policy update while preserving token-level credit assignment. section 6.2.7 section 6.2.1 section 6.2.4

A basic PPO value-learning pipeline is:

• Generate on-policy completions and obtain per-token rewards; in language-model PPO, these can include per-token KL penalties. section 6.2.7
• Compute discounted Monte Carlo returns backward from each token, resetting at terminal positions. section 6.2.7
• Regress each prediction V_t toward its return target, commonly with mean squared error and optionally clipped value predictions. section 6.2.7
• Detach A_t = G_t − V_t and use it to weight the policy-gradient loss. section 6.2.7

Figure 6.6 Value function training uses on-policy rollouts to compute targets. The model predicts \(V_t\) at each token, which is trained via mean squared error against the target return \(\hat{V}_t\). The advantage \(A_t = \hat{V}_t - V_t\) then weights the policy gradient update.

The rollout-to-target-to-advantage flow separates critic fitting from the policy update: returns supervise V_t, and prediction errors weight policy gradients. section 6.2.7

A value function is not an outcome reward model (ORM), even when both use a per-token head. An ORM predicts an immediate token-local reward or probability of correctness from fixed offline labels. A value function predicts the remaining sum of rewards from the current state and is typically trained against on-policy rollout returns. With dense correctness rewards and γ = 1, the former learns r_t while the latter learns the remaining sum from t onward. section 5.6 section 5.6.1

Choosing a critic versus a sequence baseline

PPO uses a learned value network to assign different advantages to individual tokens, incorporating per-token KL penalties before computing A_t. REINFORCE and RLOO avoid a learned critic: RLOO uses other completions for the same prompt as a leave-one-out baseline, then broadcasts a sequence-level advantage across tokens. This saves the extra model copy but loses PPO's token-level value estimates. section 6.2.4 section 6.6

How Reinforcement Learning from Human Feedback covers it

How deep it goes

Derives expected discounted return, connects it to policy gradients, and details rollout-based PPO regression without fully deriving GAE.

Examples

Uses token-level language-model states, post-KL returns, PPO critic loss, and RLOO leave-one-out baselines.

What it assumes

Assumes reinforcement-learning notions of states, actions, rewards, trajectories, and policies while defining return and discounted value.

What you can do after

Apply it via PPO-style rollout targets, value regression, and per-token advantage computation.

How it connects

Connects returns, discount factors, policy gradients, advantages, baselines, temporal-difference residuals, PPO, and reward models.

liveBookmanning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsA function that estimates the expected cumulative future reward from a state when subsequent actions follow a specified policy.

Compare how 2 books explain value function →

Related concepts

• Group Relative Policy Optimizationcontrasts with · Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled outputs or actions.
• Proximal Policy Optimizationused by · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
• Bellman equationkind of · A Bellman equation recursively relates a state or action value to an immediate reward and the expected discounted value of successor states or actions.
• outcome reward modelcontrasts with · An outcome reward model assigns a reward to the final outcome of a task or trajectory rather than to intermediate reasoning steps.
• baselineused by · A baseline is a reference value, model, condition, or performance level used to evaluate or compare measurements, outcomes, or methods.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.