Value function
A value function assigns or estimates the expected cumulative reward, utility, or quality associated with a state or action under a policy or decision rule.
Reinforcement Learning from Human Feedback
A value network is a neural network that estimates the expected value of states or actions in reinforcement learning.
- How it is used
- Supplies PPO with per-token baselines for computing advantages during language-model RL training.
- How deep it goes
- Derives expected discounted return, connects it to policy gradients, and details rollout-based PPO regression without fully deriving GAE.
- Examples
- Uses token-level language-model states, post-KL returns, PPO critic loss, and RLOO leave-one-out baselines.
- What you can do after
- Apply it via PPO-style rollout targets, value regression, and per-token advantage computation.
Read how Reinforcement Learning from Human Feedback explains value function →
Optimization Algorithms: AI techniques for design, planning, and control problems
A function that estimates the expected cumulative future reward from a state when subsequent actions follow a specified policy.
- How it is used
- Estimates expected future reward under a policy to evaluate states or actions and guide reinforcement-learning policy improvement.
- How deep it goes
- Defines state- and action-value functions, explains discounting, and works through an A2C critic implementation without a formal derivation.
- Examples
- Uses a probabilistic treasure-game return and a PyTorch critic trained for CartPole balancing.
- What you can do after
- Build it from scratch in PyTorch as a CartPole A2C critic and track its loss during training.
Related concepts
- Group Relative Policy Optimizationcontrasts with · Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled outputs or actions.
- Proximal Policy Optimizationused by · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
- Bellman equationkind of · A Bellman equation recursively relates a state or action value to an immediate reward and the expected discounted value of successor states or actions.
- outcome reward modelcontrasts with · An outcome reward model assigns a reward to the final outcome of a task or trajectory rather than to intermediate reasoning steps.
- baselineused by · A baseline is a reference value, model, condition, or performance level used to evaluate or compare measurements, outcomes, or methods.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.