A chatbot can answer confidently yet miss the document that contains the user’s crucial detail. Retrieval metrics measure whether a retrieval-augmented generation system finds relevant information from indexed data, helping protect response relevance, user satisfaction, and trust. section 6.5.2
Retrieval accuracy focuses on whether the system fetches the most relevant information for a query. Precision asks how many retrieved documents are relevant; recall asks how much of the relevant information was retrieved. RAG evaluation also uses context precision for contextual relevance and context recall for contextual coverage. section 6.5.2
Use retrieval metrics to improve and judge a RAG system:
- Tune search parameters and retrieval algorithms to improve speed, precision, and recall. section 6.5.2
- Benchmark indexing, embeddings, and prompt changes against representative user queries instead of relying on intuition. section 6.5.4
- Balance recall and precision: broad coverage can increase irrelevant results, while narrow retrieval can omit critical details. section 6.5.2
- Interpret retrieval quality alongside latency, throughput, and resource use when selecting a practical configuration. section 6.5.4
Evaluating PharmaBot
For questions such as “Can I take ibuprofen with my blood pressure medication?”, compile relevant medical sources and representative queries, then compare indexing and embedding configurations. One evaluation reports recall rising from 0.85 to 0.88, precision from 0.75 to 0.78, and F1 from 0.80 to 0.83, while average latency rises from 50 to 60 milliseconds. section 6.5.4
Table 6.4 Critical aspects influencing retrieval metrics
|
Aspect
|
Summary
|
Example
|
|---|---|---|
|
Retrieval accuracy
|
Evaluates the system’s ability to retrieve relevant information from indexed data
|
Ensuring a chatbot retrieves accurate troubleshooting guides from a large dataset
|
|
Precision and recall
|
Precision measures relevance of retrieved docs; recall measures how many relevant docs are retrieved.
|
Balancing precision and recall when retrieving product recommendations in an e-commerce chatbot
|
|
Context precision and context recall
|
Specific to RAG: context precision checks relevance; context recall checks coverage of relevant info
|
Evaluating how well a generated response in a support chatbot matches the query’s context
|
|
Parameter optimization
|
Tuning search parameters and algorithms to improve speed, accuracy, and relevance of results
|
Adjusting FAISS clusters or Elasticsearch settings to improve document retrieval for legal databases
|
|
Embedding models
|
The use of different embeddings impacts the retrieval quality by enhancing precision or recall.
|
Using BERT for precise context understanding in a legal advice chatbot
|
|
Filtering and reranking
|
Strategies to remove noise and rerank results to improve both relevance and accuracy
|
Filtering out irrelevant articles in a news aggregation chatbot and then reranking top results
|
|
Normalized Discounted Cumulative Gain (NDCG)
|
Assesses the ranking quality by considering relevance and position of retrieved documents
|
Ensuring the most relevant help articles appear at the top in a technical support chatbot
|
Compare retrieval accuracy, precision, recall, context metrics, and parameter optimization in one reference table. section 6.5.2
