liveBook · Manning

Policy

As explained in Build a DeepSeek Model (From Scratch)

A policy is a decision rule that assigns action probabilities or actions to situations encountered by a reinforcement-learning agent.

A policy is the rule an agent uses to choose actions from states. In neural reinforcement learning, model parameters determine that rule, and training changes them to make actions associated with higher long-term reward more likely. section 7.1.1

From cleaning actions to tokens

A cleaning robot observes its room, chooses whether to move, vacuum, or return to its dock, and receives feedback as the room changes. For a language model, the policy is the probability distribution over the next token; a rollout samples tokens to form a response, which can then receive a reward. section 7.1.1 section 7.2

A basic policy-gradient update adds α times the sum over sampled tokens of R ∇θ log πθ(aₖ | sₖ) to the parameters θ. Here πθ(aₖ | sₖ) is the probability assigned to sampled token aₖ in state sₖ, α is the learning rate, and R weights the update. This compact form uses raw reward; a baseline can instead make the learning signal relative to expected reward. section 7.2.2 section 7.3.2

A clipped language-model training objective distinguishes three policy roles:

  • The current policy is the trainable model. It scores generated tokens and receives the gradient update. section 7.3.1 section 7.8.4
  • The old policy produced the rollout. Its stored token log probabilities are compared with current log probabilities to form the clipped probability ratio. section 7.10 section 7.8.3
  • The reference policy is normally a frozen copy from before reinforcement learning. A KL penalty uses it as an anchor against excessive drift; it is distinct from a value model that predicts expected reward. section 7.3.1 section 7.12

In GRPO, sample several completions per prompt, score them, and normalize their rewards within the group to obtain relative advantages. Score the same generated tokens under the current and reference policies, then apply a clipped objective and an optimizer step. Unlike PPO's learned value baseline, the group-relative signal removes the need for a learned value model. section 7.12 section 7.8.4

Listing 7.4 The clipped GRPO loss with reference-policy KL
def grpo_loss(
    logp: torch.Tensor,
    old_logp: torch.Tensor,
    ref_logp: torch.Tensor,
    advantages: torch.Tensor,
    eps: float = 0.2,
    beta: float = 0.04,
) -> torch.Tensor:
    """Compute the clipped GRPO objective as a minimization loss.
 
    Shapes:
        logp, old_logp, ref_logp: [batch_size, group_size, tokens]
        advantages: [batch_size, group_size]
    """
    ratio = torch.exp(logp - old_logp)
    clipped = ratio.clamp(1.0 - eps, 1.0 + eps)
    adv = advantages.unsqueeze(-1)
    surrogate = torch.minimum(ratio * adv, clipped * adv)
    kl = logp - ref_logp
    return -(surrogate - beta * kl).mean()

The simplified PyTorch loss shows the current-to-old probability ratio, clipping, group advantages, and reference-policy KL term. A real trainer must also mask prompt and padding tokens; the displayed KL estimate is simplified. section 7.8.3

For a real trainer, store generated token IDs and old log probabilities during rollout. When rescoring, gather each generated token's log probability from the preceding model position, and mask prompt and padding tokens out of the loss. Monitor reward alongside KL: rising reward with exploding KL is a warning sign. section 7.10 section 7.12

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Develops a policy-gradient update and clipped GRPO training skeleton, while leaving generation, scoring, masking, and distributed infrastructure for a real trainer.
Examples
Illustrates the policy with a cleaning robot and sampled language-model completions scored by a math-answer verifier.
What it assumes
Builds on a pretrained language model and training pipeline while grounding reinforcement learning in an agent-environment loop.
What you can do after
Apply it via a PyTorch GRPO training skeleton, identifying where rollout generation and token scoring must be implemented.
How it connects
Connects policies to policy gradients, PPO, GRPO, value functions, rewards, and KL divergence.

Also explained in

Compare how 21 books explain policy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.