liveBook · Manning

Goodhart's Law

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Goodhart's Law states that a measure often becomes unreliable when it is adopted as an optimization target.

Goodharting occurs when an evaluation measure becomes an optimization target and stops reliably representing the underlying goal. An AI system can satisfy the evaluator while failing the user's intent: optimizing engagement may produce clickbait, while optimizing safety may produce blanket refusals. section 1.3.1 section 7.4.1

Diagnose Goodharting by checking how the evaluation signal is used:

  • Identify the desired outcome before selecting a metric; distinguish the behavior being optimized from the goal the metric is meant to represent. section 1.3.1
  • Look for gaps between score improvement and real performance, especially when an optimizer can exploit loopholes in the reward surface. section 7.4.1
  • Treat evaluation as a design constraint and use error analysis to identify failures before repeating the alignment cycle. section 1.3.1

A metric that defeats its purpose

An assistant rewarded only for user engagement may learn to write clickbait rather than helpful answers. A summarizer optimized against an automated metric can score well while humans rate its summaries below a baseline. section 1.3.1 section 7.4.1

Figure 1.10 The iterative cycle of AI engineering. Define establishes what success looks like, Evaluate measures the current system against the designed evaluation metrics, Error analysis identifies failure modes, and then Align optimizes the system using different methods. The cycle then repeats.

Use the iterative cycle to connect Goodharting with metric design, evaluation, error analysis, and alignment: define success, evaluate the system, inspect failures, optimize, and repeat. section 1.3.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains the measure-target gap, optimizer loopholes, and iterative diagnostic response, but does not provide a formal mitigation algorithm.
Examples
Uses clickbait engagement, blanket safety refusals, token-count gaming, and automated-metric summarization as failure cases.
What you can do after
Diagnose metric gaming and design evaluation criteria around desired outcomes before optimizing an AI system.
How it connects
Connects evaluation metrics with reward signals, alignment, reinforcement learning, error analysis, and human preference data.

Also explained in

Compare how 3 books explain Goodhart's Law →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.