A RAG answer can be on-topic while its retriever fills the context window with unrelated passages. Context relevance measures how much of the retrieved context can help answer the query. It evaluates retrieval, not whether the generated answer is faithful or addresses the question. section 14.2.3 section 14.0
An LLM identifies sentences in the retrieved context that could help answer the question, returning “Insufficient Information” if it finds none. The score is the number of extracted useful sentences divided by the total number of context sentences. A score of 0.3 means roughly 70% of the retrieved sentences did not help answer the query; that material consumes context-window space and can confuse the generator. section 14.2.3
The equation makes the sentence-count denominator explicit: this is a measure of useful content within what was retrieved, not a count of all relevant material in the knowledge base. section 14.2.3
A noisy reapplication search
For a question about reapplying after a credit denial, retrieval returns 15 sentences. Nine help answer the question, so context relevance is 9/15 = 0.60. General underwriting guidance may be adding noise. The same evaluation separately finds an unsupported claim in the answer: improving retrieval precision and investigating generation faithfulness are different jobs. section 14.2.4
Use the score as a diagnostic signal, not a complete retrieval verdict:
- Near-zero relevance can signal missing useful content; investigate the corpus or retrieval fallback. section 15.5.2
- A middling score with useful passages present points to noisy retrieval; examine chunking or reranking. section 14.2.4
- A high score cannot by itself establish that every needed passage was retrieved. Check ranking and context-filtering failures with complementary retrieval metrics. section 14.2.3 section 15.5.2
In RAGAS 0.4.3, `ContextRelevance(llm=llm).ascore(user_input=question, retrieved_contexts=contexts)` scores a query and its retrieved contexts without a reference answer. In production, retain the contexts needed for diagnosis, sample queries regularly, and watch trends: a gradual decline may indicate that a growing corpus is diluting retrieval quality. Compare scores before acting on small differences. section 14.2.4 section 15.6.1 section 14.7
