Goodharting occurs when an evaluation measure becomes an optimization target and stops reliably representing the underlying goal. An AI system can satisfy the evaluator while failing the user's intent: optimizing engagement may produce clickbait, while optimizing safety may produce blanket refusals. section 1.3.1 section 7.4.1
Diagnose Goodharting by checking how the evaluation signal is used:
- Identify the desired outcome before selecting a metric; distinguish the behavior being optimized from the goal the metric is meant to represent. section 1.3.1
- Look for gaps between score improvement and real performance, especially when an optimizer can exploit loopholes in the reward surface. section 7.4.1
- Treat evaluation as a design constraint and use error analysis to identify failures before repeating the alignment cycle. section 1.3.1
A metric that defeats its purpose
An assistant rewarded only for user engagement may learn to write clickbait rather than helpful answers. A summarizer optimized against an automated metric can score well while humans rate its summaries below a baseline. section 1.3.1 section 7.4.1
Figure 1.10 The iterative cycle of AI engineering. Define establishes what success looks like, Evaluate measures the current system against the designed evaluation metrics, Error analysis identifies failure modes, and then Align optimizes the system using different methods. The cycle then repeats.
Use the iterative cycle to connect Goodharting with metric design, evaluation, error analysis, and alignment: define success, evaluate the system, inspect failures, optimize, and repeat. section 1.3.1


