liveBook · Manning

Value function

A value function estimates the expected cumulative reward, utility, or quality associated with a state, action, or policy.

How 4 Manning books explain it

Applied Reinforcement Learning

A value function estimates the expected cumulative return from a state or state-action pair under a specified policy or optimal behavior.

How it is used
Uses value estimates to assess policies and guide sequential business decisions.
How deep it goes
Explains Bellman averaging and iterative evaluation, then implements policy improvement in a resource allocation problem.
Examples
Works through inventory ordering, project resource allocation, and restaurant table scheduling.
What you can do after
Build value estimates from scratch through iterative policy evaluation and use them to compare actions.

Read how Applied Reinforcement Learning explains value function →

Reinforcement Learning from Human Feedback

A value network is a neural network that estimates the expected value of states or actions in reinforcement learning.

How it is used
Uses token-level return estimates as baselines for policy-gradient updates in language-model reinforcement learning.
How deep it goes
Defines discounted value and develops Monte Carlo critic targets, masked regression, clipping, and advantages, while noting the added complexity of generalized advantage estimation.
Examples
Illustrates critic training with a PyTorch-style backward return loop, completion masks, value clipping, and detached advantages.
What you can do after
Operate and configure a PPO critic using rollout returns, completion masks, optional value clipping, and an initialization choice.

Read how Reinforcement Learning from Human Feedback explains value function →

Optimization Algorithms: AI techniques for design, planning, and control problems

A function that estimates the expected cumulative future reward from a state when subsequent actions follow a specified policy.

How it is used
Uses value estimates to judge future reward and provide a baseline for actor-critic policy updates.
How deep it goes
Defines state and action values, explains discounting, and works through a neural critic and its training updates.
Examples
Uses a probabilistic treasure-game policy and a CartPole critic to illustrate value estimates.
What you can do after
Apply it using PyTorch to train a CartPole critic alongside an A2C actor.

Read how Optimization Algorithms: AI techniques for design, planning, and control problems explains value function →

Build a DeepSeek Model (From Scratch)

A value function estimates the expected return associated with a state, action, or policy in reinforcement learning.

How it is used
Supports PPO advantage calculation by estimating expected reward from partial language-model responses.
How deep it goes
Explains learned state prediction, return-based training, advantage subtraction, and computational costs; contrasts removal through GRPO.
Examples
Uses the equation Aₖ = Rₖ − Vϕ(sₖ) and contrasts it with group-normalized rewards from four sampled completions.
What you can do after
Evaluate when PPO needs a learned critic and when GRPO can replace it with group-relative reward normalization.

Read how Build a DeepSeek Model (From Scratch) explains value function →

Also covered in

What to know first

Related concepts

Questions about value function

What is value function?

A value function estimates the expected cumulative reward, utility, or quality associated with a state, action, or policy.

Which Manning books explain value function?

Applied Reinforcement Learning, Reinforcement Learning from Human Feedback, Optimization Algorithms: AI techniques for design, planning, and control problems and Build a DeepSeek Model (From Scratch).

What should I know before learning value function?

Manning books that teach value function usually introduce Proximal Policy Optimization, reinforcement learning, reward signal and action first.

What is the difference between value function and Group Relative Policy Optimization?

A value function estimates the expected cumulative reward, utility, or quality associated with a state, action, or policy. Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled actions or outputs.

What is value function a kind of?

Value function is a kind of Bellman equation. A Bellman equation recursively relates a state or action value to an immediate reward and the expected discounted value of successor states or actions.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.