Goodhart's Law
Goodhart's Law is the principle that a measure can become less reliable as an indicator of an underlying goal when it becomes a target for optimization.
LLM Evaluation and Alignment, The Foundational Ideas
Goodhart's Law states that a measure often becomes unreliable when it is adopted as an optimization target.
- How it is used
- Frames evaluation design as an alignment problem because reward signals directly update prompts, agents, or models.
- How deep it goes
- Explains the measure-target gap, optimizer loopholes, and iterative diagnostic response, but does not provide a formal mitigation algorithm.
- Examples
- Uses clickbait engagement, blanket safety refusals, token-count gaming, and automated-metric summarization as failure cases.
- What you can do after
- Diagnose metric gaming and design evaluation criteria around desired outcomes before optimizing an AI system.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains Goodhart's Law →
Outlier Detection in Python
The principle that a measure ceases to be a reliable measure when it becomes the target of optimization.
- How it is used
- Guards outlier-detection evaluation against reusing the detector’s own measure as its test.
- How deep it goes
- Defines the principle, gives detector-specific examples, and stops short of a universal remedy beyond using different evaluation methods.
- Examples
- Uses entropy and KDE evaluation reuse, plus classifiers trained and tested against synthetic outliers.
- What you can do after
- Evaluate outlier detectors through inspection or different synthetic data instead of reusing the detection measure.
Read how Outlier Detection in Python explains Goodhart's Law →
Reinforcement Learning from Human Feedback
Goodhart's law is the principle that a measure used as an optimization target can cease to be a reliable measure of the underlying goal.
- How it is used
- Frames RLHF failures as proxy-objective exploitation rather than simple memorization or an inherent flaw in the RL algorithm.
- How deep it goes
- Explains behavioral symptoms, reward-performance trade-offs, and error sources, while leaving their precise causal origins an open research question.
- Examples
- Uses verbose answers, rare-token repetition, sycophancy, over-refusal, hedging, and repetitive phrasing as concrete failure examples.
- What you can do after
- Diagnose reward-model over-optimization by comparing proxy scores with downstream evaluations, human judgment, and KL distance.
Read how Reinforcement Learning from Human Feedback explains Goodhart's Law →
Also covered in
- AI Model EvaluationIn early access (MEAP)
- Grokking Statistics
- Building LLM Applications with DSPy
- Build High Value Software SystemsIn early access (MEAP)
- The Well-Grounded Data Analyst
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- reference modelA reference model is a fixed policy model whose output distribution provides a baseline for comparing or constraining an optimized policy.
- reinforcement learningReinforcement learning is a machine-learning paradigm in which an agent learns behavior or a decision policy through interaction with an environment and evaluative feedback, such as rewards.
- RLHFReinforcement learning from human feedback is a model-training approach that uses human evaluations or preferences as feedback to optimize model behavior.
Related concepts
- metriccontrasts with · A metric is a quantitative measure that assigns values to a property or outcome so it can be assessed, compared, or monitored.
- reward hackingequivalent to · Reward hacking is behavior that exploits weaknesses or mismatches in a reward or evaluation specification to obtain a high measured score without achieving the intended objective.
- objective functioncontrasts with · An objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
- code coveragecontrasts with · Code coverage is a measure of which portions or behaviors of a program are exercised during testing.
Questions about Goodhart's Law
What is Goodhart's Law?
Goodhart's Law is the principle that a measure can become less reliable as an indicator of an underlying goal when it becomes a target for optimization.
Which Manning books explain Goodhart's Law?
LLM Evaluation and Alignment, The Foundational Ideas, Outlier Detection in Python and Reinforcement Learning from Human Feedback.
What should I know before learning Goodhart's Law?
Manning books that teach Goodhart's Law usually introduce language model, reference model, reinforcement learning and RLHF first.
What is the difference between Goodhart's Law and metric?
Goodhart's Law is the principle that a measure can become less reliable as an indicator of an underlying goal when it becomes a target for optimization. A metric is a quantitative measure that assigns values to a property or outcome so it can be assessed, compared, or monitored.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


