Goodharting
Goodharting is the tendency of a measure to become less reliable as an indicator of an underlying goal when it becomes a target for optimization.
Outlier Detection in Python
The principle that a measure ceases to be a reliable measure when it becomes the target of optimization.
- How it is used
- Guards outlier-detection evaluation against reusing the detector’s own measure as its test.
- How deep it goes
- Defines the principle, gives detector-specific examples, and stops short of a universal remedy beyond using different evaluation methods.
- Examples
- Uses entropy and KDE evaluation reuse, plus classifiers trained and tested against synthetic outliers.
- What you can do after
- Evaluate outlier detectors through inspection or different synthetic data instead of reusing the detection measure.
Reinforcement Learning from Human Feedback
Goodhart's law is the principle that a measure used as an optimization target can cease to be a reliable measure of the underlying goal.
- How it is used
- Frames RLHF failures as proxy-objective exploitation rather than simple memorization or an inherent flaw in the RL algorithm.
- How deep it goes
- Explains behavioral symptoms, reward-performance trade-offs, and error sources, while leaving their precise causal origins an open research question.
- Examples
- Uses verbose answers, rare-token repetition, sycophancy, over-refusal, hedging, and repetitive phrasing as concrete failure examples.
- What you can do after
- Diagnose reward-model over-optimization by comparing proxy scores with downstream evaluations, human judgment, and KL distance.
Read how Reinforcement Learning from Human Feedback explains Goodharting →
Related concepts
- metriccontrasts with · A metric is a quantitative measure used to characterize or assess a property, condition, behavior, performance, or outcome.
- reward hackingequivalent to · Reward hacking is behavior that exploits weaknesses or mismatches in a reward or evaluation specification to obtain a high measured score without achieving the intended objective.
- objective functioncontrasts with · An objective function assigns values to candidate solutions, predictions, or parameter settings according to a criterion to be minimized or maximized.
- code coveragecontrasts with · Code coverage measures the extent to which a test suite exercises a program’s code elements, execution paths, or behaviors.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.