liveBook · Manning

Log probability

As explained in Build a DeepSeek Model (From Scratch)

A log probability is the logarithm of the probability assigned to an event, such as a generated token under a language-model policy.

A log probability is the logarithm of the probability assigned to an event. In language-model reinforcement learning, it records how strongly a policy assigned a generated token in its state. The gradient of this value turns reward into pressure to increase or decrease that token's probability. section 7.2.1 section 7.2.2

Use token log probabilities to connect sampled completions with policy updates:

  • Record old-policy log probabilities during rollout; the clipped ratio compares the updated policy with the policy that generated the response. section 7.10 section 7.12
  • Run current and reference policies on prompt-plus-completion sequences, gather probabilities for generated tokens, and mask prompt and padding tokens. section 7.10 section 7.8.3
  • Apply the completion's advantage across its generated tokens; clipping and reference-policy KL constrain excessively large changes. section 7.9.1

Sampling and reinforcement

After a partial answer, a policy might assign probabilities 0.42 to “has,” 0.23 to “owns,” and 0.11 to “will.” Sampling can select a lower-probability token, whose log probability is then weighted by the completion reward or advantage. section 7.2.1 section 7.9.1

Listing 7.4 The clipped GRPO loss with reference-policy KL
def grpo_loss(
    logp: torch.Tensor,
    old_logp: torch.Tensor,
    ref_logp: torch.Tensor,
    advantages: torch.Tensor,
    eps: float = 0.2,
    beta: float = 0.04,
) -> torch.Tensor:
    """Compute the clipped GRPO objective as a minimization loss.
 
    Shapes:
        logp, old_logp, ref_logp: [batch_size, group_size, tokens]
        advantages: [batch_size, group_size]
    """
    ratio = torch.exp(logp - old_logp)
    clipped = ratio.clamp(1.0 - eps, 1.0 + eps)
    adv = advantages.unsqueeze(-1)
    surrogate = torch.minimum(ratio * adv, clipped * adv)
    kl = logp - ref_logp
    return -(surrogate - beta * kl).mean()

The clipped GRPO listing exposes the implementation path: exponentiate the difference between current and old log probabilities, clip the ratio, weight by advantages, and subtract reference-policy KL. section 7.8.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Progresses from sampled-token intuition to masked tensor scoring and clipped objectives, while deferring production estimators and distributed infrastructure.
Examples
Uses sampled next-token probabilities, completion-level advantages, masked response sequences, and a PyTorch clipped-GRPO loss.
What it assumes
Assumes probability distributions and language-model next-token decisions, while grounding policy, state, action, reward, and advantage terminology.
What you can do after
Apply it via a GRPO implementation that scores generated tokens, stores rollout values, masks inputs, and computes clipped policy updates.
How it connects
Ties policy gradients to reinforcement learning, PPO, GRPO, advantages, sequence masking, and KL divergence.

Also explained in

Compare how 10 books explain log probability →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.