An objective function is a scalar target used to quantify prediction or optimization error. Training either maximizes the objective directly or minimizes a loss formed by negating it; these functions guide local parameter updates rather than guaranteeing globally desirable behavior. section 6.2.5 section 14.1.1
In policy optimization, the objective combines policy probabilities with an advantage estimate: actions better than the expected value should become more likely, while worse actions should become less likely. A zero advantage produces no policy update. section 6.3.1
Practical objective design adds structure around the basic gradient signal:
- PPO corrects distribution mismatch with an importance ratio comparing the current and data-collection policies. section 6.2.5
- Clipping the ratio to [1−ε, 1+ε] limits destructive policy changes during one update. section 6.2.5
- Language-model objectives are commonly computed per token from differences in log-probabilities, with token averaging used in practice. section 6.2.5
- Reward and cost functions can specify explicit success in finite-control domains, whereas human preferences do not reduce to one closed-form reward as easily. section 10.4
Reading a policy-gradient loss
For a positive advantage, the loss encourages the model to increase the action probability. For a negative advantage, it discourages a newly favored action. For a zero advantage, the loss is zero and the policy remains unchanged. section 6.3.1
Equation 6.28
\[\label{eq:PPO_EQN} J(\theta) = \min\left(\frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}A, \text{clip} \left( \frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}, 1-\varepsilon, 1+\varepsilon \right) A \right).\]The PPO clipped surrogate objective takes the smaller of the unclipped and clipped advantage-weighted policy ratios, preventing excessive updates while preserving useful gradient direction. section 6.2.5
