Coherence measures whether an output’s claims and reasoning fit together consistently and intelligibly. In generative systems, it captures a quality that clicks, watch time, and other engagement metrics can miss. section 8.4.3 section 9.2
Evaluate coherence with complementary signals rather than treating one metric as sufficient:
- Track immediate re-queries or repeated questions as evidence that a response failed to communicate coherently. section 8.4.3
- Use correction rate and downstream task abandonment to identify outputs users must repair or cannot use. section 8.4.3
- Combine deterministic checks with structured LLM judgment for open-ended outputs requiring coherence, relevance, factuality, tone, and safety assessment. section 9.2
Movie recommendation explanations
An LLM-based recommender may increase clicks with persuasive explanations while producing incoherent or misleading movie summaries. Evaluate explanation coherence alongside factuality, hallucination rate, novelty, retention, and satisfaction rather than relying on CTR alone. section 8.5.2

