liveBook · Manning

Proxy reward function

A proxy reward function is an observable scoring function that imperfectly approximates a less observable, delayed, or otherwise intended objective.

How 3 Manning books explain it

LLM Evaluation and Alignment, The Foundational Ideas

A proxy reward function is an imperfect objective that assigns scores correlated with, but not identical to, the intended goal.

How it is used
Frames reward design as an alignment problem where optimization can diverge from the intended goal.
How deep it goes
Explains reward hacking, metric attack surfaces, and contributing training conditions without specifying a complete mitigation procedure.
Examples
Uses boat racing, Lego stacking, video-game scoring bugs, and degraded generated text to illustrate reward exploitation.
What you can do after
Diagnose how an optimizer can exploit a metric instead of achieving the intended objective.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains proxy reward function →

Applied Reinforcement Learning

A proxy reward is an observable signal that serves as an approximation for a later or less observable desired outcome.

How it is used
Uses early observable behavior to support learning when the desired outcome arrives late or is censored.
How deep it goes
Defines the role of proxy signals and contrasts them with buffering pending outcomes, without deriving an update rule.
Examples
Uses email opens and add-to-cart events as early signals for purchases after multiple visits.
What you can do after
Apply it via early behavioral signals or buffered outcomes when direct feedback cannot update learning immediately.

Read how Applied Reinforcement Learning explains proxy reward function →

Reinforcement Learning from Human Feedback

A proxy reward is a reward signal that approximates a desired human or task objective rather than measuring it directly.

How it is used
Deploys proxy rewards in RLHF to optimize human preferences when direct measurement of chatbot usefulness is unavailable.
How deep it goes
Defines reward-model scoring and explains over-optimization, but stops short of detailed RLHF optimization choices.
Examples
Uses reward-model scores, downstream-quality curves, and RLHF training pipelines as concrete explanatory artifacts.
What you can do after
Diagnose over-optimization by comparing proxy-reward trends with downstream quality and user satisfaction.

Read how Reinforcement Learning from Human Feedback explains proxy reward function →

What to know first

Related concepts

Questions about proxy reward function

What is proxy reward function?

A proxy reward function is an observable scoring function that imperfectly approximates a less observable, delayed, or otherwise intended objective.

Which Manning books explain proxy reward function?

LLM Evaluation and Alignment, The Foundational Ideas, Applied Reinforcement Learning and Reinforcement Learning from Human Feedback.

What should I know before learning proxy reward function?

Manning books that teach proxy reward function usually introduce reinforcement learning and reward signal first.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.