Proximal Policy Optimization (PPO) is a reinforcement-learning algorithm used to optimize a language-model policy against a learned reward while keeping it close to a supervised reference model. In RLHF, it is the third stage, after supervised fine-tuning and reward-model training. section 10.8 section 10.9
A PPO fine-tuning setup coordinates:
- A policy that generates responses and a value head that estimates their expected reward. section 10.8.5
- A reference model for the KL penalty and a reward model for scoring responses. section 10.8.5
- Rollout generation, advantage computation, and policy updates; PPO reuses each rollout batch across multiple epochs. section 10.8.5 section 10.9
Medical support chatbot
For a safety-sensitive chatbot, PPO fine-tunes a supervised model using reward-model scores while a KL penalty discourages excessive divergence from the reference. Safety requirements include avoiding diagnoses, recommending professional consultation, and escalating emergencies. section 10.8 section 10.9
Listing 10.9 PPO Update — The Core Algorithm (summarized)
def ppo_update(self, samples: List[RolloutSample]) -> Dict[str, float]:
self.policy.train()
old_log_probs = torch.stack([s.log_probs for s in
➥samples]).to(self.device) #A
rewards = torch.tensor([s.reward for s in samples], device=self.device)
old_values = torch.tensor([s.value for s in samples],
➥device=self.device)
advantages = torch.tensor([s.advantage for s in samples],
➥device=self.device) #B
returns = torch.tensor([s.return_val for s in samples],
➥device=self.device)
for ppo_epoch in range(self.config.ppo_epochs): #C
use_amp = self.device == "cuda" and self.model_config
➥.dtype != torch.float32
with torch.autocast(device_type="cuda",
➥dtype=self.model_config.dtype, enabled=use_amp):
➥logits, values = self.policy(full_ids, attention_mask,
➥return_values=True)
new_log_probs = self._compute_log_probs(logits, full_ids)
ratio = torch.exp(new_log_probs
➥- old_log_probs) #D
surr1 = ratio * advantages #E
ratio_clipped = torch.clamp(
ratio,
1.0 - self.config.clip_epsilon,
1.0 + self.config.clip_epsilon,
)
surr2 = ratio_clipped * advantages #F
policy_loss = -torch.min(surr1, surr2)
➥.mean() #G
with torch.no_grad():
clipped = (ratio < 1.0 - self.config.clip_epsilon) | \
(ratio > 1.0 + self.config.clip_epsilon)
num_clipped += clipped.sum().item()
total_samples += ratio.numel() #H
probs = F.softmax(logits, dim=-1)
entropy = -(probs * torch.log(probs + 1e-8)).sum(dim=-1)
➥.mean() #I
value_pred_clipped = old_values + torch.clamp(
values - old_values,
-self.config.value_clip_epsilon,
self.config.value_clip_epsilon,
)
value_loss_unclipped = (values - returns) ** 2
value_loss_clipped = (value_pred_clipped - returns) ** 2
value_loss = 0.5 * torch.max(value_loss_unclipped,
➥value_loss_clipped).mean() #J
loss = (
policy_loss
+ self.config.value_loss_coef * value_loss
- self.config.entropy_coef * entropy
) #K
self.optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(
self.policy.parameters(),
self.config.max_grad_norm,
)
self.optimizer.step() #L
total_policy_loss += policy_loss.item()
total_value_loss += value_loss.item()
total_entropy += entropy.item()
with torch.no_grad():
new_logits = self.policy(full_ids, attention_mask)
new_log_probs = self._compute_log_probs(new_logits, full_ids)
ref_logits = self.ref_model(full_ids, attention_mask).logits
ref_log_probs = self._compute_log_probs(ref_logits, full_ids)
kl_div = (new_log_probs - ref_log_probs)
➥.mean().item() #M
if kl_div > self.config.target_kl * 1.5:
self.kl_coef *= 1.2 #N
elif kl_div < self.config.target_kl / 1.5:
self.kl_coef /= 1.2 #O
self.kl_coef = max(self.config.min_kl_coef,
➥min(self.config.max_kl_coef,
➥self.kl_coef)) #P
clip_fraction = num_clipped / total_samples if total_samples > 0 else 0
return {
"policy_loss": total_policy_loss / self.config.ppo_epochs,
"value_loss": total_value_loss / self.config.ppo_epochs,
"entropy": total_entropy / self.config.ppo_epochs,
"kl_div": kl_div,
"mean_reward": rewards.mean().item(),
"clip_fraction": clip_fraction,
}Inspect the summarized PPO update, which uses rollout samples, advantages, returns, and repeated update epochs. section 10.9





