liveBook · Manning

Context relevance

As explained in Retrieval Augmented Generation, The Foundational Ideas

Context relevance is a metric measuring how much retrieved context is useful for answering the question.

A RAG answer can be on-topic while its retriever fills the context window with unrelated passages. Context relevance measures how much of the retrieved context can help answer the query. It evaluates retrieval, not whether the generated answer is faithful or addresses the question. section 14.2.3 section 14.0

An LLM identifies sentences in the retrieved context that could help answer the question, returning “Insufficient Information” if it finds none. The score is the number of extracted useful sentences divided by the total number of context sentences. A score of 0.3 means roughly 70% of the retrieved sentences did not help answer the query; that material consumes context-window space and can confuse the generator. section 14.2.3

CR=number of extracted sentencestotal number of sentences in c(q)

The equation makes the sentence-count denominator explicit: this is a measure of useful content within what was retrieved, not a count of all relevant material in the knowledge base. section 14.2.3

A noisy reapplication search

For a question about reapplying after a credit denial, retrieval returns 15 sentences. Nine help answer the question, so context relevance is 9/15 = 0.60. General underwriting guidance may be adding noise. The same evaluation separately finds an unsupported claim in the answer: improving retrieval precision and investigating generation faithfulness are different jobs. section 14.2.4

Use the score as a diagnostic signal, not a complete retrieval verdict:

  • Near-zero relevance can signal missing useful content; investigate the corpus or retrieval fallback. section 15.5.2
  • A middling score with useful passages present points to noisy retrieval; examine chunking or reranking. section 14.2.4
  • A high score cannot by itself establish that every needed passage was retrieved. Check ranking and context-filtering failures with complementary retrieval metrics. section 14.2.3 section 15.5.2

In RAGAS 0.4.3, `ContextRelevance(llm=llm).ascore(user_input=question, retrieved_contexts=contexts)` scores a query and its retrieved contexts without a reference answer. In production, retain the contexts needed for diagnosis, sample queries regularly, and watch trends: a gradual decline may indicate that a growing corpus is diluting retrieval quality. Compare scores before acting on small differences. section 14.2.4 section 15.6.1 section 14.7

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Derives a ratio from LLM-extracted useful sentences and works a nine-of-fifteen example.
Examples
Calculates relevance for a credit-reapplication query using nine useful sentences among fifteen retrieved sentences.
What it assumes
Assumes a RAG query and retrieved context; requires no curated reference-answer dataset.
What you can do after
Apply it via RAGAS ContextRelevance to score retrieved contexts and investigate noisy retrieval.
How it connects
Connects retrieval precision to faithfulness, answer relevance, chunking, and reranking during RAG diagnosis.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.