liveBook · Manning

Policy gradient method

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A policy-gradient method optimizes a parameterized action-selection policy by estimating how changes to its parameters affect expected reward.

A policy-gradient method uses reward to update a policy. When the reward is an automated metric, the score can rise even as people judge the outputs worse. section 7.4.1

When the score misleads

Translation systems optimized against BLEU, summarizers against ROUGE, and image-captioners against CIDEr showed a similar pattern: reward scores climbed, but human ratings fell below unoptimized baselines. A summarizer trained on human preferences received higher human ratings. section 7.4.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Contrasts metric-optimized outputs with human judgments but does not detail the policy-gradient algorithm.
Examples
Compares translation with BLEU, summarization with ROUGE, and image captioning with CIDEr.
What you can do after
Understand why a rising reward score may not indicate better human-rated outputs.
How it connects
Connects reinforcement learning with reward metrics, Goodhart’s law, and human preference data.

Also explained in

Compare how 6 books explain policy gradient method →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.