liveBook · Manning

Value function

As explained in Build a DeepSeek Model (From Scratch)

A value function estimates the expected return associated with a state, action, or policy in reinforcement learning.

A value function estimates the expected return from a partial response state. In PPO, a learned value model predicts how promising the current state is, while the policy selects the next token. section 7.3.1 section 7.3.2

For rollout state sₖ, the value model Vϕ(sₖ) is trained from eventual returns. The resulting baseline converts return into an advantage: Aₖ = Rₖ − Vϕ(sₖ). section 7.3.2

A PPO-style value path typically adds:

  • A value head or separate critic with parameters ϕ, conceptually distinct from policy parameters θ. section 7.3.2
  • Value forward passes, a value loss, an optimizer path, and sometimes generalized advantage estimation. section 7.10.3
  • Additional memory, computation, and engineering complexity, which become significant for large language models. section 7.3.2

Why GRPO removes the value function

GRPO samples several completions for one prompt and normalizes their rewards. If four completions receive rewards 1, 1, 0, and 0, the group mean is 0.5; correct completions receive positive advantages and incorrect ones negative advantages. Group statistics replace the learned value baseline, while the reference model still provides KL control. section 7.4 section 7.4.3

Aₖ = Rₖ - Vϕ(sₖ)

Use the equation to connect rollout returns, learned state values, and policy-gradient advantages. section 7.3.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains learned state prediction, return-based training, advantage subtraction, and computational costs; contrasts removal through GRPO.
Examples
Uses the equation Aₖ = Rₖ − Vϕ(sₖ) and contrasts it with group-normalized rewards from four sampled completions.
What it assumes
Assumes reinforcement-learning rollouts, returns, rewards, policies, and advantages while grounding the value predictor from first principles.
What you can do after
Evaluate when PPO needs a learned critic and when GRPO can replace it with group-relative reward normalization.
How it connects
Connects reinforcement learning with PPO, policy gradients, advantage estimation, GRPO, reward signals, and reference-model KL control.

Also explained in

Compare how 4 books explain value function →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.