Relevance checking evaluates whether retrieved information matches a query’s subject and intent. In retrieval-augmented generation, weak checking can make an accurate language model produce an unusable response because the wrong context was retrieved. section 11.2.2
Wrong endpoint context
A query about the Auth API—login, logout, and validate endpoints—returned risks about deleting rooms, invalid room IDs, and deletion confirmation messages. The response looked like hallucination, but the underlying failure was selecting a room-based user story as the most relevant result. section 11.2.2
Improve relevance checking by:
- Trying different queries based on available user stories. section 11.2.2
- Comparing desired results with erroneous results. section 11.2.2
- Tweaking queries and considering advanced tooling when multiple data sources compete. section 11.2.2
