Value function
A value function estimates the expected cumulative reward, utility, or quality associated with a state, action, or policy.
Applied Reinforcement Learning
A value function estimates the expected cumulative return from a state or state-action pair under a specified policy or optimal behavior.
- How it is used
- Uses value estimates to assess policies and guide sequential business decisions.
- How deep it goes
- Explains Bellman averaging and iterative evaluation, then implements policy improvement in a resource allocation problem.
- Examples
- Works through inventory ordering, project resource allocation, and restaurant table scheduling.
- What you can do after
- Build value estimates from scratch through iterative policy evaluation and use them to compare actions.
Read how Applied Reinforcement Learning explains value function →
Reinforcement Learning from Human Feedback
A value network is a neural network that estimates the expected value of states or actions in reinforcement learning.
- How it is used
- Uses token-level return estimates as baselines for policy-gradient updates in language-model reinforcement learning.
- How deep it goes
- Defines discounted value and develops Monte Carlo critic targets, masked regression, clipping, and advantages, while noting the added complexity of generalized advantage estimation.
- Examples
- Illustrates critic training with a PyTorch-style backward return loop, completion masks, value clipping, and detached advantages.
- What you can do after
- Operate and configure a PPO critic using rollout returns, completion masks, optional value clipping, and an initialization choice.
Read how Reinforcement Learning from Human Feedback explains value function →
Optimization Algorithms: AI techniques for design, planning, and control problems
A function that estimates the expected cumulative future reward from a state when subsequent actions follow a specified policy.
- How it is used
- Uses value estimates to judge future reward and provide a baseline for actor-critic policy updates.
- How deep it goes
- Defines state and action values, explains discounting, and works through a neural critic and its training updates.
- Examples
- Uses a probabilistic treasure-game policy and a CartPole critic to illustrate value estimates.
- What you can do after
- Apply it using PyTorch to train a CartPole critic alongside an A2C actor.
Build a DeepSeek Model (From Scratch)
A value function estimates the expected return associated with a state, action, or policy in reinforcement learning.
- How it is used
- Supports PPO advantage calculation by estimating expected reward from partial language-model responses.
- How deep it goes
- Explains learned state prediction, return-based training, advantage subtraction, and computational costs; contrasts removal through GRPO.
- Examples
- Uses the equation Aₖ = Rₖ − Vϕ(sₖ) and contrasts it with group-normalized rewards from four sampled completions.
- What you can do after
- Evaluate when PPO needs a learned critic and when GRPO can replace it with group-relative reward normalization.
Read how Build a DeepSeek Model (From Scratch) explains value function →
Also covered in
What to know first
- Proximal Policy OptimizationProximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
- reinforcement learningReinforcement learning is a machine-learning paradigm in which an agent learns behavior or a decision policy through interaction with an environment and evaluative feedback, such as rewards.
- reward signalA reward signal is a value or feedback indicating the desirability, benefit, or cost associated with an action, outcome, or behavior.
- actionAn action is a choice or operation selected or performed by an actor that can affect a system, environment, or process.
Related concepts
- Group Relative Policy Optimizationcontrasts with · Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled actions or outputs.
- Proximal Policy Optimizationused by · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
- baselineused by · A baseline is a reference model, result, measurement, or level used to compare performance or assess change.
- Bellman equationkind of · A Bellman equation recursively relates a state or action value to an immediate reward and the expected discounted value of successor states or actions.
- outcome reward modelcontrasts with · An outcome reward model assigns a reward to the final outcome of a task or trajectory rather than to intermediate reasoning steps.
- policy evaluationused by · Policy evaluation is the process of estimating the expected value of states or outcomes under a fixed policy.
Questions about value function
What is value function?
A value function estimates the expected cumulative reward, utility, or quality associated with a state, action, or policy.
Which Manning books explain value function?
Applied Reinforcement Learning, Reinforcement Learning from Human Feedback, Optimization Algorithms: AI techniques for design, planning, and control problems and Build a DeepSeek Model (From Scratch).
What should I know before learning value function?
Manning books that teach value function usually introduce Proximal Policy Optimization, reinforcement learning, reward signal and action first.
What is the difference between value function and Group Relative Policy Optimization?
A value function estimates the expected cumulative reward, utility, or quality associated with a state, action, or policy. Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled actions or outputs.
What is value function a kind of?
Value function is a kind of Bellman equation. A Bellman equation recursively relates a state or action value to an immediate reward and the expected discounted value of successor states or actions.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



