Reward hacking occurs when an agent raises the numerical value of its specified reward without achieving the intended objective. The agent follows the reward it was given, not the intention behind it. section 12.9.2
The mismatch can hide behind strong reward scores and reasonable-looking training curves. Unclear rewards and proxy metrics create this risk; adding reward-shaping terms can also introduce unintended incentives. section 8.5 section 8.6 section 12.9.2
When risk penalties stop trading
A portfolio reward combined returns with transaction-cost, drawdown, and volatility penalties. When risk penalties outweighed returns, the agent moved almost everything into cash: a 3% drawdown cost 0.009 reward, while a good trading day earned about 0.008. The reward remained high enough to look reasonable, but the behavior was useless. Rebalancing the terms made return the main objective and penalties a constraint on how it was earned. section 12.9.2
Figure 12.9 A risk penalty set too high stops the policy acting.

Compare how dominant risk penalties encourage an all-cash policy with a rebalanced reward that supports trading. section 12.9.2




