liveBook · Manning

Comparison check

As explained in Effective Conversational AI

A comparison check is an evaluation of a generated answer against retrieved passages to identify unsupported content.

A generated answer can sound plausible while adding claims that the retrieved passages never support. A comparison check catches that gap before the system misleads the user. section 6.2.2

The system compares the candidate answer with retrieved passages, checking how many words and phrases overlap. A low match rate indicates weak grounding and can trigger a fallback or human transfer. section 6.2.2

A medical answer that overreaches

Retrieved passages define achalasia, dysphagia, and common vaccine side effects, but do not link booster side effects to worsening dysphagia. If the generated answer makes that unsupported claim, the system can acknowledge the limitation and connect the user with a specialist. section 6.2.2

How Effective Conversational AI covers it

How deep it goes
Measures word and phrase overlap, then deflects, falls back, or transfers when support is insufficient.
Examples
Uses a medical question about achalasia, dysphagia, booster side effects, and an unsupported causal claim.
What it assumes
Assumes retrieved passages and a generated candidate answer are available for comparison.
What you can do after
Diagnose weakly grounded answers and route users toward a fallback or human specialist.
How it connects
Connects retrieval-augmented generation with passage retrieval, answer generation, fallback responses, and human agents.
Cover of Effective Conversational AI
liveBookmanning.com
Read Effective Conversational AI on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Effective Conversational AI. The section links point to the parts of the book it draws on.