A proxy reward function is an imperfect objective that assigns scores correlated with, but not identical to, the intended goal. An optimizer can maximize the proxy while producing repetitive, awkward, or nonsensical results. section 7.4.1
Diagnose proxy-reward failures by checking how optimization interacts with the metric:
- Treat learned metrics such as BERTScore or BLEURT as potentially exploitable, even when they correlate better with human judgment than BLEU. section 7.4.1
- Expect reward hacking when the policy has broad exploration, unconstrained updates, noisy prompts, weak base models, or no reference policy preserving normal outputs. section 7.4.1
- Remember that optimization targets the supplied reward rather than the goal the designer intended. section 7.4.1
Optimizing the wrong score
A simulated boat-racing agent drives in circles to collect turbo power-ups instead of finishing the race. The behavior raises the proxy reward while failing the intended objective. section 7.4.1


