liveBook · Manning

Proximal Policy Optimization

As explained in Applied Reinforcement Learning

PPO is a policy-gradient reinforcement-learning algorithm that constrains policy updates to maintain stable improvements.

Proximal Policy Optimization (PPO) is a reinforcement-learning algorithm used to optimize a language-model policy against a learned reward while keeping it close to a supervised reference model. In RLHF, it is the third stage, after supervised fine-tuning and reward-model training. section 10.8 section 10.9

A PPO fine-tuning setup coordinates:

  • A policy that generates responses and a value head that estimates their expected reward. section 10.8.5
  • A reference model for the KL penalty and a reward model for scoring responses. section 10.8.5
  • Rollout generation, advantage computation, and policy updates; PPO reuses each rollout batch across multiple epochs. section 10.8.5 section 10.9

Medical support chatbot

For a safety-sensitive chatbot, PPO fine-tunes a supervised model using reward-model scores while a KL penalty discourages excessive divergence from the reference. Safety requirements include avoiding diagnoses, recommending professional consultation, and escalating emergencies. section 10.8 section 10.9

Listing 10.9 PPO Update — The Core Algorithm (summarized)
def ppo_update(self, samples: List[RolloutSample]) -> Dict[str, float]:
    self.policy.train()
    old_log_probs = torch.stack([s.log_probs for s in
    ➥samples]).to(self.device)  #A
    rewards = torch.tensor([s.reward for s in samples], device=self.device)
    old_values = torch.tensor([s.value for s in samples],
    ➥device=self.device)
 
    advantages = torch.tensor([s.advantage for s in samples], 
    ➥device=self.device)  #B
    returns = torch.tensor([s.return_val for s in samples],
    ➥device=self.device)
 
    for ppo_epoch in range(self.config.ppo_epochs):  #C
        use_amp = self.device == "cuda" and self.model_config
        ➥.dtype != torch.float32
        with torch.autocast(device_type="cuda",
        ➥dtype=self.model_config.dtype, enabled=use_amp):
        ➥logits, values = self.policy(full_ids, attention_mask,    
        ➥return_values=True)
 
        new_log_probs = self._compute_log_probs(logits, full_ids)
 
        ratio = torch.exp(new_log_probs 
        ➥- old_log_probs)  #D
 
        surr1 = ratio * advantages  #E
        ratio_clipped = torch.clamp(
            ratio,
            1.0 - self.config.clip_epsilon,
            1.0 + self.config.clip_epsilon,
        )
        surr2 = ratio_clipped * advantages  #F
 
        policy_loss = -torch.min(surr1, surr2)
        ➥.mean()  #G
 
        with torch.no_grad():
            clipped = (ratio < 1.0 - self.config.clip_epsilon) | \
                      (ratio > 1.0 + self.config.clip_epsilon)
            num_clipped += clipped.sum().item()
            total_samples += ratio.numel()  #H
 
        probs = F.softmax(logits, dim=-1)
        entropy = -(probs * torch.log(probs + 1e-8)).sum(dim=-1)
        ➥.mean()  #I
 
        value_pred_clipped = old_values + torch.clamp(
            values - old_values,
            -self.config.value_clip_epsilon,
            self.config.value_clip_epsilon,
        )
        value_loss_unclipped = (values - returns) ** 2
        value_loss_clipped = (value_pred_clipped - returns) ** 2
        value_loss = 0.5 * torch.max(value_loss_unclipped,       
        ➥value_loss_clipped).mean()  #J
 
        loss = (
            policy_loss
            + self.config.value_loss_coef * value_loss
            - self.config.entropy_coef * entropy
        )  #K
 
        self.optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(
            self.policy.parameters(),
            self.config.max_grad_norm,
        )
        self.optimizer.step()  #L
 
        total_policy_loss += policy_loss.item()
        total_value_loss += value_loss.item()
        total_entropy += entropy.item()
 
    with torch.no_grad():
        new_logits = self.policy(full_ids, attention_mask)
        new_log_probs = self._compute_log_probs(new_logits, full_ids)
        ref_logits = self.ref_model(full_ids, attention_mask).logits
        ref_log_probs = self._compute_log_probs(ref_logits, full_ids)
        kl_div = (new_log_probs - ref_log_probs)
        ➥.mean().item()  #M
 
    if kl_div > self.config.target_kl * 1.5:
        self.kl_coef *= 1.2  #N
    elif kl_div < self.config.target_kl / 1.5:
        self.kl_coef /= 1.2  #O
    self.kl_coef = max(self.config.min_kl_coef,
    ➥min(self.config.max_kl_coef, 
    ➥self.kl_coef))  #P
 
    clip_fraction = num_clipped / total_samples if total_samples > 0 else 0
 
    return {
        "policy_loss": total_policy_loss / self.config.ppo_epochs,
        "value_loss": total_value_loss / self.config.ppo_epochs,
        "entropy": total_entropy / self.config.ppo_epochs,
        "kl_div": kl_div,
        "mean_reward": rewards.mean().item(),
        "clip_fraction": clip_fraction,
    }

Inspect the summarized PPO update, which uses rollout samples, advantages, returns, and repeated update epochs. section 10.9

How Applied Reinforcement Learning covers it

How deep it goes
Details model components, rollouts, advantages, and policy updates, while simplifying advantage estimation for single-turn dialogue.
Examples
Uses a safety-sensitive medical support chatbot with constraints on diagnosis, consultation, emergency escalation, and empathy.
What you can do after
Apply PPO in a PyTorch and Hugging Face RLHF pipeline to fine-tune a chatbot policy using a learned reward model.
How it connects
Connects PPO to RLHF, supervised fine-tuning, reward models, KL divergence, and the actor-critic pattern.

Also explained in

Compare how 6 books explain Proximal Policy Optimization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.