A RAG system may seem improved without actually answering users more accurately. Benchmarking measures performance against defined criteria so changes can be compared and their value assessed. section 6.5.4
For PharmaBot, prepare representative health queries and a dataset, then compare indexing and embedding configurations. Track outcomes such as accuracy, relevance, latency, throughput, and resource use; choose metrics that match the system’s goals. section 6.5.4
Table 6.6 Prioritized metrics for RAG evaluation based on business objectives
|
Business objective
|
Prioritized metrics
|
Why?
|
Example
|
|---|---|---|---|
|
Customer satisfaction
|
Response accuracy
Relevance
|
Directly impacts user experience and satisfaction
|
PharmaBot providing accurate answers to users’ queries
|
|
Operational efficiency
|
Latency
Throughput
|
Ensures the system can handle high query volumes quickly
|
A customer support chatbot for a large e-commerce platform should prioritize low latency to provide quick responses during peak shopping times.
|
|
Scalability
|
Queries processed per second
Resource utilization
|
Evaluates system performance under increasing loads
|
Chatbot for a healthcare provider managing seasonal spikes in appointments and queries
|
|
Cost-effectiveness
|
CPU usage
Memory usage
|
Ensures high performance without excessive resource consumption
|
Chatbot for a non-profit organization providing 24/7 mental health support on limited funding
|
Compare business objectives with the metrics that help assess them. section 6.5.4
Table 6.7 Evaluation of PharmaBot with various configurations
|
Metrics
|
Configuration 1
|
Configuration 2
|
Configuration 3
|
|
|---|---|---|---|---|
|
Response accuracy
|
Recall
|
0.85
|
0.87
|
0.88
|
|
Precision
|
0.75
|
0.77
|
0.78
|
|
|
F1 score
|
0.8
|
0.82
|
0.83
|
|
|
Relevance
|
Mean reciprocal rank (MMR)
|
0.70
|
0.72
|
0.74
|
|
Average precision
|
0.65
|
0.68
|
0.70
|
|
|
Latency
|
Average latency (ms)
|
50
|
55
|
60
|
|
Throughput
|
Queries/second
|
20
|
18
|
16
|
|
Resource utilization
|
CPU usage (%)
|
70
|
65
|
90
|
|
Memory usage
|
8
|
7
|
12
|
See how accuracy, relevance, latency, throughput, and resource use vary across PharmaBot configurations. section 6.5.4
