liveBook · Manning

Policy gradient method

As explained in Build a DeepSeek Model (From Scratch)

A policy-gradient loss is an optimization objective that weights policy log probabilities by feedback-derived advantages to update a policy.

Policy-gradient methods update an LLM from sampled responses: tokens in successful responses become more likely, while tokens in unsuccessful ones become less likely. The method uses the log probabilities of sampled tokens, weighted by reward or advantage. section 7.1.3 section 7.2

A rollout-based update works through the following steps:

  • Sample tokens from the model’s next-token distribution; stochastic sampling enables exploration beyond the most likely token. section 7.2.1
  • Treat the generated response as a trajectory and use its outcome to weight the sampled tokens’ log probabilities. section 7.2.2
  • Apply the same completion-level advantage across its generated tokens; this coarse signal does not identify which token caused success or failure. section 7.9.1
θₜ₊₁ = θₜ + α ∑ₖ ∇θ log πθ(aₖ | sₖ) R

The simplified update makes the central operation explicit: reward scales the log-probability gradient of each sampled token. section 7.2.2

This compact rule omits practical details such as sequence masks, old-policy probabilities, and KL penalties. PPO and GRPO add stabilizing mechanisms; GRPO uses group-relative advantages, clipping, and reference-policy KL. section 7.2.2 section 7.15

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Derives a reward-weighted log-probability update and flags masks, old-policy probabilities, and KL penalties as implementation concerns.
Examples
Uses stochastic next-token choices and a verifier-scored answer of 11 to illustrate trajectory-level feedback.
What it assumes
Assumes next-token distributions and sampled actions, while mapping LLM states and actions onto reinforcement-learning terms.
What you can do after
Understand how sampled-token log probabilities and rewards form a policy update.
How it connects
Connects sparse rewards and credit assignment to baselines, advantages, PPO, GRPO, and reference-policy KL.

Also explained in

Compare how 6 books explain policy gradient method →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.