liveBook · Manning

Coherence

As explained in AI Model Evaluation

Logical coherence measures whether the claims and reasoning in an output fit together consistently and intelligibly.

Coherence measures whether an output’s claims and reasoning fit together consistently and intelligibly. In generative systems, it captures a quality that clicks, watch time, and other engagement metrics can miss. section 8.4.3 section 9.2

Evaluate coherence with complementary signals rather than treating one metric as sufficient:

  • Track immediate re-queries or repeated questions as evidence that a response failed to communicate coherently. section 8.4.3
  • Use correction rate and downstream task abandonment to identify outputs users must repair or cannot use. section 8.4.3
  • Combine deterministic checks with structured LLM judgment for open-ended outputs requiring coherence, relevance, factuality, tone, and safety assessment. section 9.2

Movie recommendation explanations

An LLM-based recommender may increase clicks with persuasive explanations while producing incoherent or misleading movie summaries. Evaluate explanation coherence alongside factuality, hallucination rate, novelty, retention, and satisfaction rather than relying on CTR alone. section 8.5.2

How AI Model Evaluation covers it

How deep it goes
Defines coherence, identifies behavioral proxies, and pairs qualitative judgment with deterministic checks without specifying a formal scoring formula.
Examples
Uses repeated questions, user corrections, abandoned tasks, and movie recommendation explanations as concrete evaluation evidence.
What you can do after
Diagnoses coherence failures through re-queries, corrections, abandonment, and structured evaluation of generated explanations.
How it connects
Links coherence with factuality, relevance, tone, safety, novelty, trust, deterministic metrics, and LLM-as-a-judge.

Also explained in

Compare how 2 books explain coherence →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.