liveBook · Manning

Reward hacking

Reward hacking is behavior that exploits weaknesses or mismatches in a reward or evaluation specification to obtain a high measured score without achieving the intended objective.

How 5 Manning books explain it

LLM Evaluation and Alignment, The Foundational Ideas

Reward hacking is behavior in which a system optimizes a reward signal by exploiting the verifier's blind spots instead of achieving the intended objective.

How it is used
Diagnoses cases where optimizing a reward proxy undermines the intended task outcome.
How deep it goes
Explains proxy exploitation through examples and mitigation choices, including verifier checks and length normalization.
Examples
Illustrates test deletion, ROUGE-optimized summaries, verbose answers, sycophancy, and a simulated boat collecting power-ups instead of racing.
What you can do after
Diagnose reward hacking by comparing reward gains with task outcomes and inspecting verifier blind spots.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains reward hacking →

Applied Reinforcement Learning

Reward hacking is the exploitation of a specified reward function in ways that increase its numerical value without achieving the intended objective.

How it is used
Flags the risk that agents optimize specified rewards rather than intended objectives, especially when rewards are unclear or proxy-based.
How deep it goes
Explains the incentive mismatch through a portfolio example and numerical comparison, then recommends validating behavior and redesigning rewards.
Examples
Illustrates risk-penalty imbalance with an all-cash portfolio, a 3% drawdown costing 0.009 reward, and a good day earning 0.008.
What you can do after
Diagnose reward hacking by checking whether optimal reward-seeking behavior matches the intended task, then redesign mismatched rewards.

Read how Applied Reinforcement Learning explains reward hacking →

Build AI Drug Discovery Pipelines

Reward hacking is a failure mode in which an optimizing system exploits weaknesses in its scoring objective to achieve high scores without satisfying the intended goal.

How it is used
Frames molecular generation as proxy optimization requiring safeguards and experimental validation.
How deep it goes
Explains the mechanism, molecular consequences, mitigation guards, and validation boundary, but defers formal policy-gradient treatment.
Examples
Uses unstable or unsynthesizable high-scoring molecules and latent reinforcement learning for targeted molecular generation.
What you can do after
Diagnose proxy failure and configure molecular-generation rewards with constraints before experimental validation.

Read how Build AI Drug Discovery Pipelines explains reward hacking →

Introduction to Generative AI, Second Edition

Reward hacking is the exploitation of imperfections in a reward specification to obtain high rewards without achieving the intended objective.

How it is used
Uses reward hacking to explain why model scores may diverge from intended quality during LLM evaluation.
How deep it goes
Defines the failure mode, illustrates it with an LLM-judge example, and connects it to adversarial and sequential testing.
Examples
Uses an appended compliance claim as an example of earning a favorable judgment without satisfying user intent.
What you can do after
Diagnose reward hacking by comparing scores with intended outcomes and testing systems across adversarial, sequential, and varied conditions.

Read how Introduction to Generative AI, Second Edition explains reward hacking →

AI Model Evaluation

A failure mode in which a model exploits weaknesses in an objective or reward specification to obtain high measured reward without achieving the intended goal.

How it is used
Uses reward hacking to test whether online model gains represent genuine product improvement rather than successful proxy exploitation.
How deep it goes
Explains the mechanism, AI-specific risk, concrete failure patterns, and five mitigation strategies, stopping short of formal optimization analysis.
Examples
Illustrates task closure, incomplete forms, chatbot verbosity, recommender clickbait, corrective user effort, and lost trust.
What you can do after
Diagnose reward hacking by comparing primary metrics with guardrails, offline stress tests, segment outcomes, and downstream product measures.

Read how AI Model Evaluation explains reward hacking →

Also covered in

What to know first

Related concepts

Questions about reward hacking

What is reward hacking?

Reward hacking is behavior that exploits weaknesses or mismatches in a reward or evaluation specification to obtain a high measured score without achieving the intended objective.

Which Manning books explain reward hacking?

LLM Evaluation and Alignment, The Foundational Ideas, Applied Reinforcement Learning, Build AI Drug Discovery Pipelines, Introduction to Generative AI, Second Edition and AI Model Evaluation.

What should I know before learning reward hacking?

Manning books that teach reward hacking usually introduce language model, policy, AI agent and objective function first.

What is reward hacking a kind of?

Reward hacking is a kind of metric gaming. Metric gaming is the manipulation or exploitation of a measure to improve its reported value without genuinely improving the underlying outcome it is intended to represent.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.