Document grounding checks whether a RAG response can be traced to source documents or passages that are relevant to the user’s query. section 6.5.4
A practical grounding evaluation follows these steps:
- Prepare representative user queries. section 6.5.4
- Generate responses with the RAG model and identify their supporting documents or passages. section 6.5.4
- Compare each source with its query, assessing relevance and coverage. section 6.5.4
- Quantify performance with a grounding score, while using precision, recall, and F1 score for response relevance. section 6.5.4
A grounding check
For each generated answer, record the passages used, compare them with the query, and score how well they support the answer. Review low-scoring cases to find relevance or coverage failures. section 6.5.4
Table 6.6 Prioritized metrics for RAG evaluation based on business objectives
|
Business objective
|
Prioritized metrics
|
Why?
|
Example
|
|---|---|---|---|
|
Customer satisfaction
|
Response accuracy
Relevance
|
Directly impacts user experience and satisfaction
|
PharmaBot providing accurate answers to users’ queries
|
|
Operational efficiency
|
Latency
Throughput
|
Ensures the system can handle high query volumes quickly
|
A customer support chatbot for a large e-commerce platform should prioritize low latency to provide quick responses during peak shopping times.
|
|
Scalability
|
Queries processed per second
Resource utilization
|
Evaluates system performance under increasing loads
|
Chatbot for a healthcare provider managing seasonal spikes in appointments and queries
|
|
Cost-effectiveness
|
CPU usage
Memory usage
|
Ensures high performance without excessive resource consumption
|
Chatbot for a non-profit organization providing 24/7 mental health support on limited funding
|
Use response accuracy and relevance when grounding supports customer satisfaction; choose evaluation metrics according to the operational objective. section 6.5.4
