A policy-gradient method uses reward to update a policy. When the reward is an automated metric, the score can rise even as people judge the outputs worse. section 7.4.1
When the score misleads
Translation systems optimized against BLEU, summarizers against ROUGE, and image-captioners against CIDEr showed a similar pattern: reward scores climbed, but human ratings fell below unoptimized baselines. A summarizer trained on human preferences received higher human ratings. section 7.4.1





