How deeply does Build a Reasoning Model (From Scratch) cover advantage function?

Defines the normalization formula, explains the stability term, and shows how the values affect gradients and training diagnostics.

Advantage function

As explained in Build a Reasoning Model (From Scratch)

A normalized per-rollout signal computed from rewards that indicates how much better or worse a rollout was than the group average.

Advantages turn per-rollout rewards into a normalized signal that says how each rollout performed relative to the group for the same prompt. They are computed by subtracting the group mean reward and dividing by the group standard deviation plus a small stability constant. section 6.7 section 7.3.1

In the GRPO pipeline, advantages are the third stage: they bridge raw rewards and the policy gradient loss. Positive values favor the actions that produced a rollout, negative values suppress them, and near-zero values contribute little. section 6.7 section 6.9

Core properties in this treatment:

Mean-centered, so the average advantage is always zero. section 7.3.1
Standard deviation near 1 suggests a well-scaled learning signal. section 7.3.1
Identical rewards collapse the signal and produce no policy update. section 7.3.1

Worked rollout example

For rewards [1., 1., 0., 0.], the computed advantages are [0.8659, 0.8659, -0.8659, -0.8659]. The positive pair is treated as better than average, and the negative pair as worse than average. section 6.7 section 7.3.1

How Build a Reasoning Model (From Scratch) covers it

Cover of Build a Reasoning Model (From Scratch)

liveBook manning.com

Read Build a Reasoning Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Reinforcement Learning from Human FeedbackAn advantage function estimates how much better an action is than the policy's expected value at a state.
liveBook manning.com

Optimization Algorithms: AI techniques for design, planning, and control problemsAn advantage is the estimated improvement in expected future reward from taking an action in a state compared with the average action in that state.

Compare how 3 books explain advantage function →

Related concepts

Group Relative Policy Optimizationpart of · Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled outputs or actions.
policy gradient methodused by · A policy gradient method is a reinforcement-learning technique that uses gradients of expected return to optimize the parameters of a policy.
generalized advantage estimationproduced by · Generalized advantage estimation estimates policy advantages by exponentially weighting temporal-difference errors across multiple time scales, trading bias against variance.
reward signaluses · A reward signal is an evaluative feedback value representing the desirability or payoff of an action, outcome, or state.
varianceaddresses · Variance is the expected squared deviation of a random variable from its mean, measuring the spread of its distribution.

This explainer was generated by Manning's knowledge graph from the text of Build a Reasoning Model (From Scratch). The section links point to the parts of the book it draws on.